Debugging with GDB(felix-knorr.net)
felix-knorr.net
Debugging with GDB
https://felix-knorr.net/blog/using_gdb_directly.html
90 comments
I'd also recommend "Give me 15 minutes and I'll change your view of GDB" [1] by Greg Law at CppCon 2015. As someone who wasn't very familiar with GDB, I learned a lot from seeing it used by someone skilled. [1]: https://youtu.be/PorfLSr3DDI
One way to have up arrow go back in history in TUI mode is to press "Ctrl-X O" which "switches focus" between TUI source view and command window.
Is that capital X and O? So you need to hold shift as well?
No, it's the emacs keybind: C-x o
Emacs switch window keybinding..
The command ‘focus cmd’ does the same thing.
I like GDB, and use it a lot, but it always feels like it's teetering on the brink of being a pile of bugs. I manage to crash it semi-regularly, and there are dozens of little quality-of-life peccadilloes that make things annoying before working out a .gdbinit for yourself. It's like using Vim without a .vimrc (:
I don't envy the maintainers' jobs, and I just gripe from the sidelines, but starting from scratch it's too bad it's such an intimidating experience, because it really is a great tool. And I say this as someone who has used MSVC's debugger (which I've often seen applauded as the best) for some decades, and I still prefer GDB now that I'm past the significant initial learning curve.
I don't envy the maintainers' jobs, and I just gripe from the sidelines, but starting from scratch it's too bad it's such an intimidating experience, because it really is a great tool. And I say this as someone who has used MSVC's debugger (which I've often seen applauded as the best) for some decades, and I still prefer GDB now that I'm past the significant initial learning curve.
To me coming from GDB, MSVC 's debugger seems like nothing but a flashy toy. It cannot do any of the things I need a debugger to do beyond stepping and setting breakpoints.
Real Windows programmers use windbg, which is comparable to GDB. Even better built-ins.
But the MSVC dbg UI also has its advantages.
Still writing lots of .gdbinit scripts though.
Still writing lots of .gdbinit scripts though.
Can you give some examples? I’ve never used a debugger for more than stepping and setting breakpoints
I've used Java's debugger to do "live" UI editing on Android for around a decade now. Far in advance of first-party support / live-updating things being popular.
Just set a timer to call back to the main thread every 100ms or so. When you get to the point in an application where you want to make changes, put a breakpoint in that callback. When it pauses, pop open the expression editor and start changing the UI. Make some views, adjust some values, etc.
When you want to see the results, hit "continue"; it'll release the main thread for about 100ms (more than enough time to re-layout and render), then you'll be back in the debugger for another round of changes. You can hack up a full, polished-looking and often working UI in a couple minutes, and then just print the whole tree to get a snapshot your work.
That same strategy works in basically any language with GDB support, in many, many UI frameworks, if the connection is sophisticated enough to call functions.
Just set a timer to call back to the main thread every 100ms or so. When you get to the point in an application where you want to make changes, put a breakpoint in that callback. When it pauses, pop open the expression editor and start changing the UI. Make some views, adjust some values, etc.
When you want to see the results, hit "continue"; it'll release the main thread for about 100ms (more than enough time to re-layout and render), then you'll be back in the debugger for another round of changes. You can hack up a full, polished-looking and often working UI in a couple minutes, and then just print the whole tree to get a snapshot your work.
That same strategy works in basically any language with GDB support, in many, many UI frameworks, if the connection is sophisticated enough to call functions.
A proof that you never learned to use that toy.
How does the adult version of GDB show threads and parallel stacks and shader debugging?
Or better yet, how do adults do hot code reload on GDB?
How does the adult version of GDB show threads and parallel stacks and shader debugging?
Or better yet, how do adults do hot code reload on GDB?
GDB will gladly show you all the threads' stacks.
Couldn't tell you about shaders. Haven't needed hot code reload.
Couldn't tell you about shaders. Haven't needed hot code reload.
In a primitive text view like 1970's, letting the use draw on paper the relationship between them, or which tasks are mapped into which thread.
So a more powerful debugger that offers stuff beyond what you are aware of in computing, while GDB hasn't be able to do so in 40 years, is a toy.
So a more powerful debugger that offers stuff beyond what you are aware of in computing, while GDB hasn't be able to do so in 40 years, is a toy.
for example?
GDB is great. I definitely recommend checking out watchpoints as well, a very useful tool for monitoring how a variable changes over time.
GDB also has many good plugins - pwndbg has tons of features and UI improvements over stock GDB.
https://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.ht...
https://github.com/pwndbg/pwndbg
GDB also has many good plugins - pwndbg has tons of features and UI improvements over stock GDB.
https://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.ht...
https://github.com/pwndbg/pwndbg
Using vanilla GDB is painful. As a bit of a shameless plug I recommend you check out GEF[1]. It's a large python script that extends GDB to make it a lot better to use. Notably it shows a lot of the state automatically every time the inferior stops. It's oriented around reversing and exploit development, but it definitely doesn't have to be used that way.
[1] https://github.com/hugsy/gef
[1] https://github.com/hugsy/gef
I'll also plug pwndbg[1] here.
Like Gef it greatly extends the utility of GDB, and is oriented around exploit development.
[1] https://github.com/pwndbg/pwndbg
[1] https://github.com/pwndbg/pwndbg
I'll make a shameless plug here as Pwndbg maintainer - apart from the FEATURES.md file describing features, you can see some of them detailed on those slides: https://docs.google.com/presentation/d/1mDfA_27DtLUkOaTZ8U9k...
It looks really awesome. Thanks.
This anti-pattern is a bit of a red flag for me though:
This anti-pattern is a bit of a red flag for me though:
bash -c "$(curl -fsSL http://gef.blah.cat/sh)"I sorta agree with you but only because it's an http site which allows easy MITM attacks . Otherwise, you're going to be downloading a shell script and executing it. You are trusting your system to a blind executable either way. Same with using pip, choco, rpm, brew, dnf, etc.
Except with dnf you need signed rpms or to have explicitly disabled gpg checks to install unsigned packages from a 'trusted' source. Ie, much harder to MITM.
I've never understood why this gets so much hate.
You are downloading and running their python script anyway. Why do you trust their software but not their install script?
The fact that it is hosted on http is suboptimal, but it does redirect to an https github page in this case
You are downloading and running their python script anyway. Why do you trust their software but not their install script?
The fact that it is hosted on http is suboptimal, but it does redirect to an https github page in this case
the difference includes -- a physical computer you use and care about, need to trust, with a lot of setup and your own efforts involved, may be a single-point endpoint with value and/or state, which is by definition vulnerable versus an ephemeral VM or Docker that is repeatedly created and destroyed in moments, has almost no lasting value since a new one is only a minute away, and has no personal value or customization.
For me it is because it normalizes bad practice. There is a reason we have packages and signatures.
It is not as bad as:
It is not as bad as:
curl something | sh
How is running a script from a downloaded package any better than running a script from a download script?
The script can change any time, and maybe even just for some people without leaving a trace.
Updating a package in a OS distribution repository requires that it is signed, and spread to all the mirrors (it is unlikely that an attacker controls all of them). This makes it more likely to be discovered and hard to do targeted attacks without leaving a trace.
If it does happen that you are compromised using:
I run local mirrors of everything I use (with snapshots after updating the mirror), so I at least have a history of the software that I ran to do forensics on. I could not do that if I ran scripts directly from the Internet into a shell.
For example: Something suddenly trying to connect to the Internet (or doing DNS lookups) from one of my segregated systems might prompt me to investigate. Even if you are not so "paranoid" as many would call it (I would call it being diligent), you still get the benefit of others being "paranoid".
So making sure (via packages, mirrors and signatures) that we all get the same "version" of software is important for security.
Updating a package in a OS distribution repository requires that it is signed, and spread to all the mirrors (it is unlikely that an attacker controls all of them). This makes it more likely to be discovered and hard to do targeted attacks without leaving a trace.
If it does happen that you are compromised using:
bash -c "$(curl -fsSL http://gef.blah.cat/sh)"
I run local mirrors of everything I use (with snapshots after updating the mirror), so I at least have a history of the software that I ran to do forensics on. I could not do that if I ran scripts directly from the Internet into a shell.
For example: Something suddenly trying to connect to the Internet (or doing DNS lookups) from one of my segregated systems might prompt me to investigate. Even if you are not so "paranoid" as many would call it (I would call it being diligent), you still get the benefit of others being "paranoid".
So making sure (via packages, mirrors and signatures) that we all get the same "version" of software is important for security.
How can you detect that it’s going to a pipe?
The shell reads a command and then executes it. It doesn't buffer the whole script into memory.
The directions for manual installation are listed directly after that. The first page of the documentation even explains how to install it directly from the repo. Also, it's funny that "curling to shell" is what crosses the line. As if any of the installation options even approach the risk of appending .gdbinit with over ten thousand lines of unaudited python, especially in a privileged environment like a debugger. Curling to shell is not particularly dangerous than the alternatives.
I love that anti-pattern. It gets people thinking. It’s not like anyone is gonna download that script, find out what the script downloads, and then read every line of the whole project and its dependencies to see if one of them installs a rootkit.
It certainly is. I don't install it that way, or recommend it be installed that way, but others do. You can take that up with the project owner :)
[deleted]
I can't believe no one has mentioned `gdb-dashboard` [1] yet! I use it extensively. [2]
Beyond that, I have recently learned how to write custom pretty printers for GDB. This saves a lot of screen space. I should probably update [2] soon with those new techniques.
GDB is powerful, useful, and after getting my start in IDE debuggers, including Visual Studio, I struggle whenever I have to go back.
[1]: https://github.com/cyrus-and/gdb-dashboard
[2]: https://gavinhoward.com/2020/12/my-development-environment-a...
Beyond that, I have recently learned how to write custom pretty printers for GDB. This saves a lot of screen space. I should probably update [2] soon with those new techniques.
GDB is powerful, useful, and after getting my start in IDE debuggers, including Visual Studio, I struggle whenever I have to go back.
[1]: https://github.com/cyrus-and/gdb-dashboard
[2]: https://gavinhoward.com/2020/12/my-development-environment-a...
I haven't seen anyone demonstrate using gdb via CLI in the way you can show off using vim proficiently and have people say "I get it".
In the reverse engineering space there's some folks that use gef. but maybe that's because linux folks dont have olydbg?
would love to see a livestream of someone using gdb in the context of real world engineering. SHOW me the productivity wins here. every article and conference talk about gdb is just handwavey hypotheticals about what you could do, but in practice I just see people moving through basic debugging incredibly slowly
In the reverse engineering space there's some folks that use gef. but maybe that's because linux folks dont have olydbg?
would love to see a livestream of someone using gdb in the context of real world engineering. SHOW me the productivity wins here. every article and conference talk about gdb is just handwavey hypotheticals about what you could do, but in practice I just see people moving through basic debugging incredibly slowly
As a CLI power user I can say that I don't think CLI gives much productivity wins over UI/IDE debugging. However, it lets you script things, display things that your IDE wouldn't normally display for you and do some automatic or semi-automatic analysis. Also, as you said indeed there is no ollydbg, or rather x64-dbg equivalent on Linux (which is what people use nowadays on Windows as ollydbg is not developed anymore afaik).
Also even if you use GDB in IDE, you still may need to fall back to the embedded GDB console as e.g. some classes are missing pretty printers or the IDE UI does not allow you to investigate navigate things properly. For example when you have a structure field that is void* and is casted to one or another type depending on the context, it is much easier to use the console to investigate it.
Also even if you use GDB in IDE, you still may need to fall back to the embedded GDB console as e.g. some classes are missing pretty printers or the IDE UI does not allow you to investigate navigate things properly. For example when you have a structure field that is void* and is casted to one or another type depending on the context, it is much easier to use the console to investigate it.
> but in practice I just see people moving through basic debugging incredibly slowly
What are you comparing this to? A person skilled with an IDE?
What are you comparing this to? A person skilled with an IDE?
There are no goal-posts that are both sides will find acceptable in these discussions.
As a general rule i have found it almost impossible to 'change' someones mind, you can influence an undecided person, but changing someones mind is generally a waste of time.
Unless you have financial incentive to do this, you're going to be running this at a loss, let the 'fixed mind' party take the hit of being wrong.
As a general rule i have found it almost impossible to 'change' someones mind, you can influence an undecided person, but changing someones mind is generally a waste of time.
Unless you have financial incentive to do this, you're going to be running this at a loss, let the 'fixed mind' party take the hit of being wrong.
single video of someone starting, stopping, and setting breakpoints across hundred+ file programs or during a regular debug task please
I came across this writeup: https://infosecwriteups.com/pwndbg-gef-peda-one-for-all-and-...
It describes how to configure GDB with the GEF, PEDA and PWNDBG plugins.
(I've only really used gdb-peda, would be great to get some time to learn the others)
It describes how to configure GDB with the GEF, PEDA and PWNDBG plugins.
(I've only really used gdb-peda, would be great to get some time to learn the others)
I can really recommend debugging with command-line gdb (inside a text editor, of course; we're not barbarians). The transcript that lets you look back at the past of your debug session really adds a new dimension to the debug experience that a plain IDE integration just does not offer.
(Time-travelling debuggers are OK, too, I guess)
(Time-travelling debuggers are OK, too, I guess)
What do you mean by transcript? I used gdb a while ago and it's mostly similar to a regular terminal application with readline.
By transcript, I mean every command and every response in the debug session. Usually, in the beginning, I do not know yet what I'm looking for, and set breakpoints at points that may be interesting, and when I reach the breakpoints, I look at variables. Later, when I reach the point where things have gone wrong, I can look at variable values now, but also at the result of every previous query. This helps me answer questions like: was this object here already in that queue over there when the previous request came in?
I can also attach the debug session to an issue in order for others or future me to understand what was happening then. In a purely GUI-driven debugger, I can copy&paste a stack trace of the final point, but the history is lost.
I can also attach the debug session to an issue in order for others or future me to understand what was happening then. In a purely GUI-driven debugger, I can copy&paste a stack trace of the final point, but the history is lost.
I've used scripts + logging in the past:
https://sourceware.org/gdb/onlinedocs/gdb/Logging-Output.htm...
Maybe that is what OP means by a transcript.
https://sourceware.org/gdb/onlinedocs/gdb/Logging-Output.htm...
Maybe that is what OP means by a transcript.
Just like I was doing with XEmacs in 1995...
There is so much a modern IDE debugging session is capable of.
There is so much a modern IDE debugging session is capable of.
What do you mean “inside a text editor”? What’s the workflow that you describe like?
I think it just means inside your editor's terminal, where there's usually some modest integrations like it detecting a source/line reference and either jumping you there or at least making it clickable.
VSCode for sure does this.
VSCode for sure does this.
I cannot for the life of me get gdb to debug a remote in VS Code. The debugger runs, and breakpoints work and integrate, but the console is dead. '-exec print foo' does nothing.
Time travel to the UNIX world of early X Windows days.
Now picture XEmacs and Emacs, running gdb as subprocess, with a little pointed finger showing the current line and a stop sign for breakpoints.
The lower buffer shows the usual gdb repl and output.
Now picture XEmacs and Emacs, running gdb as subprocess, with a little pointed finger showing the current line and a stop sign for breakpoints.
The lower buffer shows the usual gdb repl and output.
Time travel to the late 70s and use the lisp machine debugger.
I always felt the drive towards mouse-driven tools in the PARC world (InterLisp/SmallTalk/CedarMesa) was actually a regression because of the loss of history ("how the hell did I get here?")
I always felt the drive towards mouse-driven tools in the PARC world (InterLisp/SmallTalk/CedarMesa) was actually a regression because of the loss of history ("how the hell did I get here?")
They also had a proper REPL, and the reason I stay with Java and .NET ecosystems, is because they are the closest we can get back to the Xerox experience, after the UNIX divergence.
Still available with DDD and various plugins to vim/emacs/vscode
GDB has a really nice TUI mode that helps a lot while debugging.
This! I recently discovered this as well and it makes gdb feel like a modern-day debugger like windbg.
`C-x a` to toggle it on
`C-l` to re-draw screen when stdout gets spit out and messes up TUI
`C-p/n` for scrolling gdb history
`up/down` for scrolling code
`C-x a` to toggle it on
`C-l` to re-draw screen when stdout gets spit out and messes up TUI
`C-p/n` for scrolling gdb history
`up/down` for scrolling code
Why not just open a proper IDE like QtCreator ? You can just use it as a gdb UI ("Debugging > Debug external application" or something like that) if you want and you get all that + editable text, a tree view for variables...
Also `C-x o` switches focus between the code and command windows, which changes function of up/down arrows (scrolling/command history).
Which, at least for me, needs an initial "C-x C-r" before it starts functioning via the normal "C-x C-a". I think there was some old bug about it, but I was never able to get to the bottom of it.
But agreed, TUI mode is handy (:
But agreed, TUI mode is handy (:
After several decades of using GUI-based debugging in MSVC++ and CLion, I can't bring myself to regress back to command-line debugging. And sadly the GUI tools for GDB are abysmal.
Recently I had to debug some bootloader code in QEMU, which provides a remote GDB port. I tried every GDB GUI and they all either (A) assumed that I would provide source code to the target code (it's a binary blob bootloader, which I don't have code for) or (B) flat out didn't work. I ended up using TUI mode, which has it's own problems (messes up my terminal when I disconnect/reconnect the GDB port)
Recently I had to debug some bootloader code in QEMU, which provides a remote GDB port. I tried every GDB GUI and they all either (A) assumed that I would provide source code to the target code (it's a binary blob bootloader, which I don't have code for) or (B) flat out didn't work. I ended up using TUI mode, which has it's own problems (messes up my terminal when I disconnect/reconnect the GDB port)
I'm surprised to see people praising GDB so much here. Working on Chromium, using GDB was just hopeless. Difficult to get working, incredibly slow to the point of apparently hanging, practically no support for multi process debugging, absolutely atrociously bad GUI frontends. Visual Studio's debugger works great in the same scenario. And windbg is there if you hate GUIs.
I have little hope for the state of Linux debugging in the future if people really think the current situation with GDB and its frontends is good or even acceptable.
I have little hope for the state of Linux debugging in the future if people really think the current situation with GDB and its frontends is good or even acceptable.
Having spent my entire career inside of Visual Studio (nee VS Code), this strikes me as a quaint relic from a time past.
Can it be powerful? Probably. But the dev experience looks terrible.
Can it be powerful? Probably. But the dev experience looks terrible.
Microsoft makes an excellent debugger. One advantage gdb has, though, is that you can attach it to almost any process over a terminal. Another (that I've seen used to great effect, but not since a long time ago) is that it's very straightforward to build up powerful libraries of debugging scripts—things like dumping or following custom data structures, which can be really useful for core file analysis and what not.
VS debugger can do none of the things I look to a debugger to do.
Dev experience is fixable in many ways, but wholesale lack of capability is not.
Dev experience is fixable in many ways, but wholesale lack of capability is not.
VSCode is really frustrating lately, I debug all day long and the IDE just randomly shits a brick, or hangs, or just stalls, the vim plugin just loses its shit once in a while do ruining your undo's.
MSVC was (is) by far much better overall (I only pull it out now a days to analyze crash dumps though). Heck in vscode you can't really see variables in C++ or examine memory, or set memory change break points, but really 90% of developing doesn't require any of that, so vscode just keeps growing in use.
I do love me some GDB though, when I'm developing on *nix anyway.
MSVC was (is) by far much better overall (I only pull it out now a days to analyze crash dumps though). Heck in vscode you can't really see variables in C++ or examine memory, or set memory change break points, but really 90% of developing doesn't require any of that, so vscode just keeps growing in use.
I do love me some GDB though, when I'm developing on *nix anyway.
> VS debugger can do none of the things I look to a debugger to do.
Such as? I'm genuinely curious here. I know of a bunch of things that VS can't do on Windows that requires resorting to windbg (which is a terrible experience compared to VS's debugger), but what are the things you look for in a debugger that it can't do at all? Are we talking user land debugging or kernel?
Such as? I'm genuinely curious here. I know of a bunch of things that VS can't do on Windows that requires resorting to windbg (which is a terrible experience compared to VS's debugger), but what are the things you look for in a debugger that it can't do at all? Are we talking user land debugging or kernel?
I use gdbserver to attach gdb to a process on the target machine, running gdb on my dev box where the sources are. In a previous life, I would use a similar capability to debug code in an embedded target through a serial port, with a tiny stub in the target program to manipulate target memory and execution. Nowadays you do that through a USB port.
When I set a breakpoint, I can make it conditional on an expression run against program data, and have it run a series of debugger commands at each stop, and then maybe continue.
Scrolling back through the transcript to see how values have changed at each stop, and running expressions on e.g. differences between such values.
All simple things.
When I set a breakpoint, I can make it conditional on an expression run against program data, and have it run a series of debugger commands at each stop, and then maybe continue.
Scrolling back through the transcript to see how values have changed at each stop, and running expressions on e.g. differences between such values.
All simple things.
You can definitely do most if not all of that with WinDBG (which is distinct from the Visual Studio builtin debugger). The problem is there doesn't seem to be as much resources around learning how to use it. There are a ton of old hats at MS who know how to perform all sorts of wizardry with WinDBG.
sometimes gdb is what you got. cant always have visual studio when debugging something in a server.
The beginning was promising. I expected something for non-vim users. As a heavy VS user, I tried some Linux IDEs like KDevelop and Code::Blocks and, AFAIR, they integrated with GDB to resemble VS debugging process. For me personally, just pressing F5 (a hew hundreds of times a day) is a bit less cumbersome than running dgb program, start, etc. )
I think you mean that VS is emulating the Unix gdb GUI debugging process. No doubt some standards have cross pollinated.
can you be more clear what you're referring to?
gdb has been mauled by Apple in macOS in the name of security
Maybe I am spoiled but I cannot bear the nagging about code-signing it yourself
Maybe I am spoiled but I cannot bear the nagging about code-signing it yourself
GDB doesn’t work anyways even after you codesign it because it’s not actively maintained.
Nonsense. Of course it's maintained. Used it code signed on macOS for years. With MacPorts, the proper installer. Not the kids toys.
It's not, and if you've actually tried to point it at programs where codesigning is necessary (i.e. native macOS programs) you'll find that GDB fails to attach to processes properly because the code is racing on a port set.
cgdb can be helpful if you want a bit more visual experience: http://cgdb.github.io
Catching up with Turbo C++ for MS-DOS development experience.
https://www.geeksforgeeks.org/debugging-in-turbo-cpp/
https://www.geeksforgeeks.org/debugging-in-turbo-cpp/
Is this better or more featureful than just gdb --tui ?
It has the virtue of working. (Maybe I just didn't know about the needed initial C-x C-r?)
There is a gap where GDB changed terminal interaction protocol and (on Debian) left cgdb behind, so I had to build my own copy of cgdb from source. Otherwise, smooth sailing.
There is a gap where GDB changed terminal interaction protocol and (on Debian) left cgdb behind, so I had to build my own copy of cgdb from source. Otherwise, smooth sailing.
It seems to be more comfortable to use (for example you can see the list of sources and search in it vim like), but may be it's subjective or I didn't explore built in options enough.
If you like gdb, I recomend to check out rr too. Reversible debugging is just a superpower.