My Favorite Debug Ever(ryan.hypnoticocelot.com)
ryan.hypnoticocelot.com
My Favorite Debug Ever
http://ryan.hypnoticocelot.com/post/136617949294/my-favorite-debug-ever
23 comments
ElectricFence is great, and I recall having a similar feeling to the OP when I first used it because many of the described issues were also new to me at the time. It's a great experience :)
If you're on Solaris or a Solaris-derived OS, you also have the libumem and watchmalloc libraries that can help you out. I have used libumem to great effect in the past.
It's been a while, but I anecdotally recall that Solaris is more stingy with its mallocs than Linux. I used to compile and run my C projects on Solaris as a first pass in my search for memory errors.
If you're on Solaris or a Solaris-derived OS, you also have the libumem and watchmalloc libraries that can help you out. I have used libumem to great effect in the past.
It's been a while, but I anecdotally recall that Solaris is more stingy with its mallocs than Linux. I used to compile and run my C projects on Solaris as a first pass in my search for memory errors.
Is using GDB and Electric Fence to find a normal buffer overflow bug really interesting enough to warrant a blog post these days, let alone a HN story?
I guess it might be interesting for all the people who, like the author, are Java and PHP programmers and have no experience with C?
I guess it might be interesting for all the people who, like the author, are Java and PHP programmers and have no experience with C?
The fact that the author wrote a blog post about it, and the fact that someone found it interesting enough to share, are iron-clad proof that these things are obviously interesting enough to write about.
The fact that he pursued the cause of the bug even when things got difficult or strange for him was the essence of his real "hacker" experience. Was it a simple buffer overflow? Maybe, but he learned a lot getting his hands dirty
In my experience, using GDB and any kind of memory debugging tool is all new to the majority of C and C++ programmers, so I'd expect them to find it interesting too.
I'm trying to read this comment in the most kind voice that I can muster, and it still comes across as rather mean. The article may not be useful to you, but it was well written, quite funny in parts, and useful to enough people to make the first page.
> I guess it might be interesting for all the people who, like the author, are Java and PHP programmers and have no experience with C?
Maybe it isn't snark, but that reads like it, and it seems unconstructive. Why knock down someone who's learning by denigrating his background? What's the harm in someone sharing something interesting (to him) that he learned, whether or not you think he should have known it already? It's easy enough not to read it, especially on a random blog and even on HN, if you don't want to share his experience.
Maybe it isn't snark, but that reads like it, and it seems unconstructive. Why knock down someone who's learning by denigrating his background? What's the harm in someone sharing something interesting (to him) that he learned, whether or not you think he should have known it already? It's easy enough not to read it, especially on a random blog and even on HN, if you don't want to share his experience.
I’m mostly trying to understand why this is deemed interesting, and not to snark. I mean, sure, anything voted up is interesting enough, but I thought that this was below the minimum level of knowledge a C programmer ought to have. It’s like if there was a story on the front page of HN about debugging HTTP using “telnet”.
But there's loads of not-C-programmers who might still work on segfaults from time to time.
For example I am absolutely able to use gdb, but have never heard of Electric Fence. (But I had heard of AddressSanitizer mentioned above.)
For example I am absolutely able to use gdb, but have never heard of Electric Fence. (But I had heard of AddressSanitizer mentioned above.)
Maybe not for people who do this kind of things almost on daily basis anyways.
It does have educational value: help to give more people pointers, short tutorial how to attack these issues. The more people can do this, the less bugs we will have around.
We need more people with these skills. It's all the time harder to hire these people nowadays.
It does have educational value: help to give more people pointers, short tutorial how to attack these issues. The more people can do this, the less bugs we will have around.
We need more people with these skills. It's all the time harder to hire these people nowadays.
Yes.
Usually the toughest things to debug is when stack is overwritten. Worse, if it's stack and heap. Not my favorite things to debug...
Here's what I usually do in such cases:
Call stack is unreliable at that point. You'll see puzzling things like function calls with impossible parameters, etc. If things just don't make any sense, it's better to map all code paths that can lead to the crashing EIP/RIP (hopefully valid pointer to the instruction that caused the crash). Check EIP/RIP if it's in some rep movsd (= potentially inlined memcpy, check ECX (RCX) rep counter, EDI (RDI) rep pointer), or if the execution is in some runtime library code such as memset, memcpy, etc. similar. The next thing is to make sense of the call stack manually, if there are portions not overwritten, but what stack walk couldn't resolve. Of course it pays to take a look around stack otherwise as well, for signs of overwrite and contents of the overwrite.
It's also possible a pointer to stack object leaked at some point and the crash occurs at completely different part of code than where it actually segfaulted. Or some runtime structure was corrupted, like heap. Sometimes you can find those by just inspecting and guessing struct/object shape and values near stack pointer ESP (RSP).
If the bug can be reproduced, memory breakpoints, logging (especially if multithreaded, but watch out for blocking I/O from logging), tools for debugging memory corruption (valgrind, compiler paranoid mode, etc.) etc, even mapping some pages unreadable and unwritable. It can take a while to find the actual bug.
If reproduction is not possible, good luck. Better spend some quality time with memory hex view, disassembler, trying to locate registers and stack values that might contain pointers, etc. It might take a while to find the issue...
Here's what I usually do in such cases:
Call stack is unreliable at that point. You'll see puzzling things like function calls with impossible parameters, etc. If things just don't make any sense, it's better to map all code paths that can lead to the crashing EIP/RIP (hopefully valid pointer to the instruction that caused the crash). Check EIP/RIP if it's in some rep movsd (= potentially inlined memcpy, check ECX (RCX) rep counter, EDI (RDI) rep pointer), or if the execution is in some runtime library code such as memset, memcpy, etc. similar. The next thing is to make sense of the call stack manually, if there are portions not overwritten, but what stack walk couldn't resolve. Of course it pays to take a look around stack otherwise as well, for signs of overwrite and contents of the overwrite.
It's also possible a pointer to stack object leaked at some point and the crash occurs at completely different part of code than where it actually segfaulted. Or some runtime structure was corrupted, like heap. Sometimes you can find those by just inspecting and guessing struct/object shape and values near stack pointer ESP (RSP).
If the bug can be reproduced, memory breakpoints, logging (especially if multithreaded, but watch out for blocking I/O from logging), tools for debugging memory corruption (valgrind, compiler paranoid mode, etc.) etc, even mapping some pages unreadable and unwritable. It can take a while to find the actual bug.
If reproduction is not possible, good luck. Better spend some quality time with memory hex view, disassembler, trying to locate registers and stack values that might contain pointers, etc. It might take a while to find the issue...
Thanks for sharing this! I could wish for some tool use (gdb) screenshots, though really the inspiration to learn is valuable enough.
A next read in this vein might be Bug Hunter's Diary by Tobias Klein: https://www.nostarch.com/bughunter
A next read in this vein might be Bug Hunter's Diary by Tobias Klein: https://www.nostarch.com/bughunter
Great article! I had a similar experience, a Java programmer turned Linux Kernel programmer. The first think I remember doing was debugging a kernel panic using crash. It is not doubt challenging for Java/PHP programmers to debug C coredumps/KP, when there is pointer reference involved. But at the same time I learned a lot, specially memory management.
I wonder how many of the young so called "programmers" would be able to solve this. It's a pity that one can call himself a "developer" without having the tiniest idea how a computer actually runs, what a process, a thread or a pointer is...
We all stand on the shoulder of giants. I'm pretty sure when the first languages appeared, some people said things like "It's a pity that one can call himself a "developer" without having the tiniest idea how a transistor actually runs, what a perfored card is..."
The problem is: what happens when there is no one any more who remembers how the basics work?
The banking and healthcare industry already has this problem because there is a huge amount of decades old legacy stuff still in use and they have to pay through their noses for expertise.
The banking and healthcare industry already has this problem because there is a huge amount of decades old legacy stuff still in use and they have to pay through their noses for expertise.
I'd say it's not the same kind of basic. We still need people that know how to manually manage memory.
We don't need them as much though. But at some point, Sun needs people to develop the JVM, and its memory management algorithm. The "how this works" is well documented in academic papers too.
Obviously, as you mention, the less there are people that are capable to do X, the more they cost. But it's not the same as "totally forgetting something".
Obviously, as you mention, the less there are people that are capable to do X, the more they cost. But it's not the same as "totally forgetting something".
I've already seen both sides.
Some low level people don't know how to do full-stack web development. Some full-stack web development people have no idea about memory management and stuff. etc...
You need all sorts of people to be productive.
Some low level people don't know how to do full-stack web development. Some full-stack web development people have no idea about memory management and stuff. etc...
You need all sorts of people to be productive.
[deleted]
I'm ok with those industries having to pay (some of) their workers more.
How many of the not-so-young so called "programmers" would be able to solve this.
As a second option I'm using valgrind which can find uninitialized memory (and doesn't require recompile of all libs as MemorySanitizer does).