Automatically download symbols and sources in GDB(sourceware.org)
sourceware.org
Automatically download symbols and sources in GDB
https://sourceware.org/elfutils/Debuginfod.html
20 comments
I strongly encourage developments like that. It’s a shame that Linux and macOS didn’t adopt something like Microsoft’s symbol server. This project could become something like this.
If you have an installation like this you can also plug it into Sentry as symbol server and it picks it up.
If you have an installation like this you can also plug it into Sentry as symbol server and it picks it up.
"Good enough" solutions were already in place for many years, that's why a symbol server was not a matter of urgency. Did you know about the following?
• A package manager can be configured to install the corresponding debuginfo/debugsource packages, this is literally one menu toggle. It's off by default to save disk space, as most users don't need them.
• A desktop environment crash handler (e.g. drkonqi) offers to install the missing debuginfo/debugsource packages and then produces a complete stacktrace.
• gdb shows the command for installing the missing debuginfo/debugsource packages it needs.
• A package manager can be configured to install the corresponding debuginfo/debugsource packages, this is literally one menu toggle. It's off by default to save disk space, as most users don't need them.
• A desktop environment crash handler (e.g. drkonqi) offers to install the missing debuginfo/debugsource packages and then produces a complete stacktrace.
• gdb shows the command for installing the missing debuginfo/debugsource packages it needs.
In Opensuse (release, haven't used Tumbleweed) I found it easy to install debuginfo and debugsource packages.
In Ubuntu I found it slightly confusing to locate debug symbol packages. I understood (or misunderstood) that there are 2 options where to find debug symbols and it just depends on the package maintainer how it's done. I did not find how to install debugsource at all.
By doing debugsource I mean the source that has all the patches applied exactly as during the build and is in the same path as the symbols suggest.
In Ubuntu I found it slightly confusing to locate debug symbol packages. I understood (or misunderstood) that there are 2 options where to find debug symbols and it just depends on the package maintainer how it's done. I did not find how to install debugsource at all.
By doing debugsource I mean the source that has all the patches applied exactly as during the build and is in the same path as the symbols suggest.
I'm not aware of those solutions. Very cool! Are they specific to one distro (for example, Fedora)? How do I get gdb to show me the command to installing missing packages?
One downside to those solutions, they only support debugging on the same machine running the executable. If I want to debug a crash dump from a machine different from my own I might not be able to install the packages.
One downside to those solutions, they only support debugging on the same machine running the executable. If I want to debug a crash dump from a machine different from my own I might not be able to install the packages.
> Are they specific to one distro
no
> How do I get gdb to show me the command
With the "run" command. You get a message "Missing separate debuginfos, use:" and then debuginfo-install, zypper etc. depending on the distro.
no
> How do I get gdb to show me the command
With the "run" command. You get a message "Missing separate debuginfos, use:" and then debuginfo-install, zypper etc. depending on the distro.
None of what we have is good enough unfortunately. If you have a random crash from a customer you’re out of luck.
You load up the debug info and get all file/line info you need, or you debug the core file.
With the Microsoft tooling, you debug a crash dump and your IDE automatically pulls in the symbol data from your symbol server and then automatically checks out the appropriate revisions from your config management system, because the references are embedded into the symbol files at build time.
It’s a much better mechanism than anything I’ve looked at in the Linux space (caveat: I have not been developing C++ for a while).
I was so impressed I started looking at what it would take to implement for the gcc/gdb toolchain but discovered Microsoft held a patent and gave up.
It’s a much better mechanism than anything I’ve looked at in the Linux space (caveat: I have not been developing C++ for a while).
I was so impressed I started looking at what it would take to implement for the gcc/gdb toolchain but discovered Microsoft held a patent and gave up.
Yep, in case of Linux the 'it's OSS. Download the tar.gz configure --enable-debug and deal with it' attitude hampered the development and use of something like a Symbol Server.
Microsoft has had it since at least 2005 - that's the first Visual Studio release I remember using it with - so it might have exited before that. It's things like these that separate MS from all others - Engineering culture of having to give 2 shits about other people - OEMs, developers, corporate IT, partners and having the chops to effectively address it at scale within their Engineering teams.
Microsoft has had it since at least 2005 - that's the first Visual Studio release I remember using it with - so it might have exited before that. It's things like these that separate MS from all others - Engineering culture of having to give 2 shits about other people - OEMs, developers, corporate IT, partners and having the chops to effectively address it at scale within their Engineering teams.
What are you talking about? Linux distros come with debug info packages.
Yeah I know - that's a more recent thing however and the attitude in the early days was what I was referring to.
Oh and the debuginfo packages used to be and still are huge in size. Not sure if anything has changed most recently but the concept of lazy loading symbols from remote and storing only what you need locally is not something you can do with debuginfo packages.
Oh and the debuginfo packages used to be and still are huge in size. Not sure if anything has changed most recently but the concept of lazy loading symbols from remote and storing only what you need locally is not something you can do with debuginfo packages.
Oh yes, the size is killing it. Our applications are not that big so the bandwidth use isn't completely insane, but I see the problem.
Apple runs something similar internally, as it can correlate debug symbols to binaries by checking the UUID the linker sticks in it. Unfortunately, it’s not generally available…
There used to be a hack where you could write your own UUID-to-dSYM script, name it "dsymForUUID", and then lldb would assume it was Apple's internal magic thing and use it. Fortunately lldb seems to have improved since those days, and has less magical options available:
http://lldb.llvm.org/use/symbols.html
http://lldb.llvm.org/use/symbols.html
Any chance that Linux package managers will integrate this for their -dbg packages? I’d rather not fiddle with downloading those along with -src, untarring it, telling GDB where to find the code…
This was originally built by RedHat and they are adding support for Fedora's package manager, which should make debugging in future versions of Fedora an amazing experience. The ability to debug any application on your machine with no effort.
Debuginfod does support debian packages as well (integrating with -dbg pacages), but it does not support src packages. The documentation mentions that this is tricky.
Debuginfod does support debian packages as well (integrating with -dbg pacages), but it does not support src packages. The documentation mentions that this is tricky.
Very small. I asked before and the consensus was that nobody wants to maintain it. There was a project before (Fedora’s dark server) which died.
That said, debian started indexing build ids in the repos so at least for dwarf files there is a barebones index now. Sadly not for the companion executables.
That said, debian started indexing build ids in the repos so at least for dwarf files there is a barebones index now. Sadly not for the companion executables.
The debug server kinda builds upon the dbg/src packages. So all the maintainers of the distributions have to do is maintain the compatibility of the dbg/src package structure (which looks like happening for the RPM format) and to host a debug server in addition to the existing package servers.
Arch may do this as they still haved figured out how distribute debug packages.
[0] https://github.com/edolstra/dwarffs/blob/master/README.md