Making a low level Linux debugger, part 3: our first program(blog.asrpo.com)
blog.asrpo.com
Making a low level Linux debugger, part 3: our first program
https://blog.asrpo.com/making_a_low_level_debugger_part_3
28 comments
It seems you've mastered some dark art unknown to an entire industry, you should totally write a book
Meanwhile I'll rely on the smelly old debugger to dig me out of threading weirdness, memory corruption, resource leakages, other people's bugs, and to provide a window into a combinatorially massive input-dependent state my feeble mind simply can't fathom while writing tests for a tiny corner of a typical system that in aggregate amounts to millions of LOC.
Meanwhile I'll rely on the smelly old debugger to dig me out of threading weirdness, memory corruption, resource leakages, other people's bugs, and to provide a window into a combinatorially massive input-dependent state my feeble mind simply can't fathom while writing tests for a tiny corner of a typical system that in aggregate amounts to millions of LOC.
> threading weirdness, memory corruption, resource leakages, other people's bugs
Just as an FYI, most of these are theoretically eliminated or at least drastically mitigated by a switch from mutable state to immutable, purely-functional data structures. The cost of course being somewhat increased memory consumption and a bit more slowness (or a lot, depending on the algorithm).
If you ever get a chance to work with the BEAM VM, it's pretty tight.
Just as an FYI, most of these are theoretically eliminated or at least drastically mitigated by a switch from mutable state to immutable, purely-functional data structures. The cost of course being somewhat increased memory consumption and a bit more slowness (or a lot, depending on the algorithm).
If you ever get a chance to work with the BEAM VM, it's pretty tight.
Nah, gradparent is not a wizard, just someone who has spent a lot of time in a single relatively simple environment that is mostly under their control. It's a beautiful thing when it happens. Most aren't so lucky.
+1 to the above, it's just a different environment. I haven't used a debugger at my current (mostly Python) job at all, whereas at a previous job in C++, I was using gdb practically every day.
He is taking after Linus Torvalds, who resisted adding a kernel debugger for many years. Linus used console spew, PC speaker beeps, and storing a hash of file/line into the PC's time/date.
... who resisted adding a kernel debugger for the explicit purpose of keeping casuals out of his zen garden, per your link below.
For people who don't have a hoard of elite developers banging down their gates to contribute code for free, I suspect the economics of being such a Prima Donna are not as attractive.
For people who don't have a hoard of elite developers banging down their gates to contribute code for free, I suspect the economics of being such a Prima Donna are not as attractive.
That's less not needing a debugger by design, more how challenging it is to debug an operating system.
Those last two (speaker and clock) yes, but otherwise no. Linus Torvalds really doesn't like using a debugger and he'd rather you didn't use one either. His words:
https://yarchive.net/comp/linux/debuggers.html
https://yarchive.net/comp/linux/debuggers.html
He's not wrong.
I still maintain that debuggers are a design smell.
I still maintain that debuggers are a design smell.
I find it depends a lot on my level of understanding of the project I'm working on. If I wrote 20% or more of the code, or if I was involved in the design, etc., when there is a problem I know just where to look and can get there quickly.
For example, if I'm working on a Python script under 1000 lines I would never use a debugger. I might use wireshark or linux system tools if something was really wrong, but usually the trackback or program output is sufficient.
I've also tried to debug an Arduino project where I had a lot less knowledge of how the system works, and if I wasn't able to attach a debugger I don't want to know how long it would have taken me to find the problem (calling a null function pointer causes an interrupt that didn't have any handler).
I've also tried to debug other people's projects in high level languages, without a debugger where after taking about an hour to figure it out based on pure reason, realizing it would have been very quick and easy to find with a debugger.
I think if you look at what a program is doing, and at that point you can't form a testable hypothesis as to what is happening, at that point a debugger will certainly be faster if you know how to use it.
For example, if I'm working on a Python script under 1000 lines I would never use a debugger. I might use wireshark or linux system tools if something was really wrong, but usually the trackback or program output is sufficient.
I've also tried to debug an Arduino project where I had a lot less knowledge of how the system works, and if I wasn't able to attach a debugger I don't want to know how long it would have taken me to find the problem (calling a null function pointer causes an interrupt that didn't have any handler).
I've also tried to debug other people's projects in high level languages, without a debugger where after taking about an hour to figure it out based on pure reason, realizing it would have been very quick and easy to find with a debugger.
I think if you look at what a program is doing, and at that point you can't form a testable hypothesis as to what is happening, at that point a debugger will certainly be faster if you know how to use it.
> I find it depends a lot on my level of understanding of the project I'm working on. If I wrote 20% or more of the code, or if I was involved in the design, etc., when there is a problem I know just where to look and can get there quickly.
Well, I mean... Ever wonder why that is? Your understanding of the various states the program can get into is fresh in your mind. Come back to it in a year after not looking at it all that time and suddenly you're relying hardcore on your unit test suite to contain that same knowledge in "automated proof" form... OR you start spending a lot of time in the debugger to "re-understand" the system, slowly and manually... Or both.
If you need to use a debugger, it's because you are in some unexpected state. With full understanding comes full control and in those cases, no, you need neither a debugger NOR a test suite, but since a test suite on well-written code already identifies many possible state failures, I've found that leaning on that instead of a debugger works out better in the end.
Well, I mean... Ever wonder why that is? Your understanding of the various states the program can get into is fresh in your mind. Come back to it in a year after not looking at it all that time and suddenly you're relying hardcore on your unit test suite to contain that same knowledge in "automated proof" form... OR you start spending a lot of time in the debugger to "re-understand" the system, slowly and manually... Or both.
If you need to use a debugger, it's because you are in some unexpected state. With full understanding comes full control and in those cases, no, you need neither a debugger NOR a test suite, but since a test suite on well-written code already identifies many possible state failures, I've found that leaning on that instead of a debugger works out better in the end.
How did you use a debugger on Arduino? I've been using Serial.print("WTF") more than I'd admit.
This is similar to what I did: https://learn.adafruit.com/proper-step-debugging-atsamd21-ar... . That is for that specific platform, not sure if there is something similar for the other versions.
I used the J-Link but I wanted to do it on Linux, so I used the JLink GDB Server along with the GDB build that comes with the Arduino IDE. It probably takes me a good 5 minutes to get everything set up, but the basic stuff works: breakpoints, stack trace, reading and writing memory locations.
The setup is a little convoluted, I'm not sure if there is a better way, but here are my notes:
I used the J-Link but I wanted to do it on Linux, so I used the JLink GDB Server along with the GDB build that comes with the Arduino IDE. It probably takes me a good 5 minutes to get everything set up, but the basic stuff works: breakpoints, stack trace, reading and writing memory locations.
The setup is a little convoluted, I'm not sure if there is a better way, but here are my notes:
Procedure for running w/ debugger attached.
1. disconnect debugger's usb
2. connect device via usb
3. arduino IDE program device
4. identify elf, set up gdb with elf.
5. plug in debugger
6. run jlink gdb server
7. attach gdb to jlink server "target remote:2331"
8. "monitor reset", "continue" to re-run target from beginning.
9. observe device is recognized by Arduino IDE
10. open Arduino IDE serial monitor to interact with device.
Now, Ctrl+C in GDB halts device, and stack etc. can be inspected, and
then it can continue running with "cont".
To rerun the device with the same program without redoing everything,
go to step 8 and reset the device. and continue from there.
To find .elf on linux:
==================================================
find /tmp -name sketch_name.ino.elf 2>/dev/null
gdb location:
==================================================
/home/user/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-gdb
JLink setup command:
==================================================
JLinkGDBServer -device ATSAMD21G18A -if SWD -speed 4000
gdb attachment:
==================================================
(gdb) target remote:2331Thanks for the info. I did buy a knockoff J-Link but haven't got around to trying it yet.
I've only used the Arduino Mini/Micro so far (with Atmega328 or Atmega32u4). Is it possible to debug on those?
I've only used the Arduino Mini/Micro so far (with Atmega328 or Atmega32u4). Is it possible to debug on those?
Not sure, I haven't used that platform personally, there is probably a debugging interface to the chip but it will probably require different hardware/software.
I have been an embedded developer working quite close to the metal. Over the years, I have noticed that the further you get away from the metal, the less reliant you are on the debugger. In projects past, I have worked with MathLab developers who are deploying their code on a multicore CPU and they have no need to have a debugger (in the traditional sense) in their tool set.
Why do you think debuggers are so necessary at the metal when they aren't further above the metal?
When debugging code that are close to the metal such as device drivers or interrupt service routines, the code's behavior is very tightly coupled to the metal.
For example, the interrupt service routine that your debugging could not reacting fast enough and you need to find out why? Priority of the said interrupt is correctly configured and it is blocked by a higher priotiy interrupt. This type of behaviour difficult to track down if a debugger is not used.
For example, the interrupt service routine that your debugging could not reacting fast enough and you need to find out why? Priority of the said interrupt is correctly configured and it is blocked by a higher priotiy interrupt. This type of behaviour difficult to track down if a debugger is not used.
[deleted]
This is exactly what I was thinking, but couldn't phrase it quite as nicely. I never feel the need to reach for pdb when writing Python, but C on microcontroller? I need my gdb.
I don't have that much to add to the other replies here. I used to never use pdb in Python for many years (not knowing or forgotten it exists) and doing perfectly well. Then I started using post-mortem debugging and finding/fixing bugs got so much faster. Basically, I'm using the computer to do the simulations I previously ran in my head (by evaluating things here and there). But that's just one point of data.
Another debugger I now recommend people (in general) to look at is Squeak Smalltalk's debugger which allows a form of "top down" programming where undefined functions are called and filled in at runtime. The debugger/editor in this post borrows a little bit from that idea in spirit. Part 3 in particular shows how it can be used even when they are no bugs.
About being off-topic, well, no-one has made another top level comment and by the time of your first post it was already 5 hours in. Some discussion is definitely better than none. So thank you for inciting that. If it burried some more on-topic discussion, that would be unfortunate though.
Another debugger I now recommend people (in general) to look at is Squeak Smalltalk's debugger which allows a form of "top down" programming where undefined functions are called and filled in at runtime. The debugger/editor in this post borrows a little bit from that idea in spirit. Part 3 in particular shows how it can be used even when they are no bugs.
About being off-topic, well, no-one has made another top level comment and by the time of your first post it was already 5 hours in. Some discussion is definitely better than none. So thank you for inciting that. If it burried some more on-topic discussion, that would be unfortunate though.
Ha, thanks for that!
I find that I don't need a debugger until I do. People are always asking me what debugger I use for go and I tell them I've gone months without feeling the need for them.
However every once in a while I need to use gdb + remote debugging features to figure out a particularly weird runtime issue and I'm glad it's there.
Then I can often take the odd values I see in the debugger and start putting them into a test or a minimal repro.
However every once in a while I need to use gdb + remote debugging features to figure out a particularly weird runtime issue and I'm glad it's there.
Then I can often take the odd values I see in the debugger and start putting them into a test or a minimal repro.
This is the ideal use case IMHO
Having access to a debugger when you need it is a necessity. Rampant use of it can, debatably, be seen as a crutch...but if you need it and you don't have it, you're going to have an extremely bad week.
Now I'm sure that kernel-level stuff wasn't designed like this, and probably won't be for some time (I'm hoping Rust changes that story, but not a low-level coder), so debuggers may still be necessary- just making a point that it seems possible to design in such a way that the need for one is drastically reduced, if not eliminated. Or at least, that's how it seems to me in high-level-language-land (things may differ at the assembly and C levels).
EDIT: I apologize for OT, probably not the ideal place to start a "are debuggers really necessary if you design code correctly?" discussion, after all this is part 3 of a series, my bad