The Sad State of Debug Performance in C++(vittorioromeo.info)
vittorioromeo.info
The Sad State of Debug Performance in C++
https://vittorioromeo.info/index/blog/debug_performance_cpp.html
110 comments
To be fair, std::unordered_map is also very slow in release mode.
It'd maybe be nice to be able to mark which code gets optimized. If I could make all std as "don't debug this and keep it optimized" I would. I know some debuggers have the option to at least not step into certain code that that doesn't solve the optimization issue
You don't have to compile every file with the same flags. You can vary them with each invocation of the compiler, so trivially get what you want at the translation unit level.
And get equivalent of ODR errors.
I really doubt flags like -g or -O if going to cause those issues.
As others have mentioned, you can accomplish the equivalent by using a release build with pragmas to turn off optimizations in certain sections of code.
Isn't it just a matter of /D_ITERATOR_DEBUG_LEVEL=0?
I think that won't work because all objects limited together have to use the same debug iterator level.
But you can go the other direction: start with release mode (make a separate build profile if you like) and turn on all debug features except debug iterators (and the debug CRT).
But you can go the other direction: start with release mode (make a separate build profile if you like) and turn on all debug features except debug iterators (and the debug CRT).
Second recommendation for robin_hood's map. I've used it as well, excellent performance compared to the default implementation in GCC.
How do you see the key/value pairs with robinhood in debug mode? I could never find them.
> debug iterators
MSVC then, I presume.
MSVC then, I presume.
The problem is that C++ is full of so called "zero-cost-abstractions" that are only zero cost if you ignore:
1. Compile time
2. Non-optimized builds
Stuffing everything into the Standard Library using lots of template magic does has its upside, but debug performance is one of the big downsides.
Another one are really clunky interfaces, like std::variant. This should really have been a language feature, not a library features.
1. Compile time
2. Non-optimized builds
Stuffing everything into the Standard Library using lots of template magic does has its upside, but debug performance is one of the big downsides.
Another one are really clunky interfaces, like std::variant. This should really have been a language feature, not a library features.
The phrase "zero-cost abstraction" is a lie anyway. It's all relative to how much compiler optimization happens. If you actually executed the C++ specification step by step, the code would be dog slow, because there are so many implicit conversions, constructions, inlining, constant folding, etc. We could quibble about how much static resolution qualifies (template expansion, overload resolution, etc--that's executing an inflated in-memory IR, and not the original source code.) Not to mention that all the rest of the compiler backend is working hard to get good machine code, and cannot possibly produce "optimal" machine code, so there is an unmeasurable cost just because of the limitations of any one particular compiler backend. (And those things matter, too!)
If you implemented any language following the spec in a step by step manner they would be dog slow, because any usable spec has to be very explicit in every step required for every operation.
All those conversions that you believe only happen if you implement the spec “step by step” always happen.
Actual implementations of languages then implement the explicitly described semantics, in whatever way they feel is most efficient.
All those conversions that you believe only happen if you implement the spec “step by step” always happen.
Actual implementations of languages then implement the explicitly described semantics, in whatever way they feel is most efficient.
Bah. Compiling Java to bytecode consists of a lot of static resolution steps and then a fairly straightforward, simple translation. Single-stepping the bytecodes will absolutely produce the same results as the optimized JIT code and there are few expressions that generate more than 1 to a handful of bytecodes.
Compile time and non-optimized builds aren't what you want to consider when you care about code that's going to run across thousands of cores or ultra low latency work - those are the cases where you should be considering C++ in the first place.
I've been using C++ since 1993. I still don't understand how I've written all this code and never once found myself using, or wishing I could use std::move.
Presumably my code has been less than optimally efficient or something, because it seems that most people talking about "modern" C++ view it as absolutely central to the language.
Presumably my code has been less than optimally efficient or something, because it seems that most people talking about "modern" C++ view it as absolutely central to the language.
> Presumably my code has been less than optimally efficient or something
Yep, it was either less efficient than it could be, or more clunky, or incorrect.
I remember back when boost was still a thing, everyone using boost::shared_ptr indiscriminately. Refcounting has a very nontrivial cost to it if implemented correctly.
Things like rvalue refs and std::move allow to properly implement the concept of "ownership transfer", obviating the need for costly ref counting in the majority of cases.
Yep, it was either less efficient than it could be, or more clunky, or incorrect.
I remember back when boost was still a thing, everyone using boost::shared_ptr indiscriminately. Refcounting has a very nontrivial cost to it if implemented correctly.
Things like rvalue refs and std::move allow to properly implement the concept of "ownership transfer", obviating the need for costly ref counting in the majority of cases.
I've been using boost::shared_ptr for 22+ years. My code has no concept of ownership transfer, ever. The reason for shared_ptr<T> in our codebase is not to move ownership around, but to ensure correct lifetimes. For examples, objects in the model cannot die until the view has dropped references to them.
> to ensure correct lifetimes
Refcounted pointers don't make that guarantee. They just guarantee that if they are incorrect they will not deallocate (== will leave garbage) and thus never result in a use-after-deletion.
Most of the time they are fine but they can't handle circular structures. This is why C++ has the weak_ptr.
Refcounted pointers don't make that guarantee. They just guarantee that if they are incorrect they will not deallocate (== will leave garbage) and thus never result in a use-after-deletion.
Most of the time they are fine but they can't handle circular structures. This is why C++ has the weak_ptr.
No use-after-deletion is precisely what is required. And yes, we use weak_ptr too, for some things.
The alternative is "no language-level references, any time you want object "Foo", go look it up in some table". We opted not to do that.
The alternative is "no language-level references, any time you want object "Foo", go look it up in some table". We opted not to do that.
Yup, that's bad. Unless you actually have a complicated ownership scheme or shared state across multiple threads, unique ownership is preferable on all accounts. At the point where
1) your object graph is complex enough to warrant the ubiquitous use of shared pointers;
2) you can afford the performance hit
it's better to use an actual garbage collector, even if it's bolted-on unreal engine-style.
With shared_ptr-style ref counting, every dtor is potentially a bomb waiting to go off causing a cascading effect. It's actually less predictable than a GC that you can run manually at fixed intervals or whenever a convenient opportunity arises.
1) your object graph is complex enough to warrant the ubiquitous use of shared pointers;
2) you can afford the performance hit
it's better to use an actual garbage collector, even if it's bolted-on unreal engine-style.
With shared_ptr-style ref counting, every dtor is potentially a bomb waiting to go off causing a cascading effect. It's actually less predictable than a GC that you can run manually at fixed intervals or whenever a convenient opportunity arises.
>It's actually less predictable than a GC that you can run manually at fixed intervals or whenever a convenient opportunity arises.
Not really. It's difficult to reason about, but it's entirely deterministic, unlike a tracing GC.
Not really. It's difficult to reason about, but it's entirely deterministic, unlike a tracing GC.
when you cross a certain level of complexity, the deterministic nature becomes a Turing tarpit of sorts: yes, it's "deterministic", but the number of factors affecting the behavior is so large and their interactions so complex that it might as well be "non-deterministic" -- it's impractical to reason about in detail.
I disagree. The big point of the determinism is that you can actually trivially control when deallocation occurs. In fact even if you don't want to manually reason about there are tools which can graphically show you when objects are deallocated.
I'll take the "almost non-deterministic" over the non-deterministic. Computers are very good at reasoning about complex deterministic systems.
...and a garbage collector is one of those. It's not magic fairy dust, it's still software that works on the same processing unit as your program, and obeys the same rules. Computers are finite automata, they can't beget true randomness, so any software system that is isolated to a single machine is "deterministic", if you have enough context.
>Computers are finite automata, they can't beget true randomness
"Non-deterministic" in this context means that the behavior of the program is not deducible from the state of the current thread of execution, or more generally, from the state of the system being analyzed. If another thread modifies the current thread's memory, that modification is non-deterministic because you couldn't have foreseen it by following any pointer that's reachable by the current thread. If a cosmic ray hits a computer chip and flips a bit, that bitflip is non-deterministic; you couldn't have predicted that that bit would be flipped by looking at any part of the computer.
In other words, an effect is non-determistic with respect to a causal chain if it's causally unrelated to it.
A tracing GC is non-deterministic for two reasons. First, it causes effects (namely, pauses) in all threads, even those that never allocate any memory. Second, it makes the lifetime of objects unpredictable. Thread A can allocate an object and then drop it, and when that object will be destroyed depends on the behavior of the entire system, not just thread A's. RAII doesn't exist in GC'd languages (except through using or try-finally hacks).
Reference counting is completely deterministic. If a thread holds the last reference to an object you can predict with certainty when that object will be released and which objects will also be released as a consequence, before actually releasing the object. You don't need to know what any other thread is doing to make this prediction.
Crucially, if you find that sometimes a reference counting program spends too long releasing an object graph, all you need to do to debug it is to run the program again under the same conditions and break when it reaches that point. Try doing the same with a garbage collector.
"Non-deterministic" in this context means that the behavior of the program is not deducible from the state of the current thread of execution, or more generally, from the state of the system being analyzed. If another thread modifies the current thread's memory, that modification is non-deterministic because you couldn't have foreseen it by following any pointer that's reachable by the current thread. If a cosmic ray hits a computer chip and flips a bit, that bitflip is non-deterministic; you couldn't have predicted that that bit would be flipped by looking at any part of the computer.
In other words, an effect is non-determistic with respect to a causal chain if it's causally unrelated to it.
A tracing GC is non-deterministic for two reasons. First, it causes effects (namely, pauses) in all threads, even those that never allocate any memory. Second, it makes the lifetime of objects unpredictable. Thread A can allocate an object and then drop it, and when that object will be destroyed depends on the behavior of the entire system, not just thread A's. RAII doesn't exist in GC'd languages (except through using or try-finally hacks).
Reference counting is completely deterministic. If a thread holds the last reference to an object you can predict with certainty when that object will be released and which objects will also be released as a consequence, before actually releasing the object. You don't need to know what any other thread is doing to make this prediction.
Crucially, if you find that sometimes a reference counting program spends too long releasing an object graph, all you need to do to debug it is to run the program again under the same conditions and break when it reaches that point. Try doing the same with a garbage collector.
> Reference counting is completely deterministic. If a thread holds the last reference to an object you can predict with certainty when that object will be released and which objects will also be released as a consequence, before actually releasing the object. You don't need to know what any other thread is doing to make this prediction.
You're contradicting your own definition of determinism here. Other threads may be holding references to the same object, and unless you know what the other threads are doing, you _cannot_ predict when an object is going to be released just by looking at the execution of a single thread.
You need to look at the overall behavior of the application, with the complex object graph spanning multiple threads.
> all you need to do to debug it is to run the program again under the same conditions
this is much easier said than done.
You're contradicting your own definition of determinism here. Other threads may be holding references to the same object, and unless you know what the other threads are doing, you _cannot_ predict when an object is going to be released just by looking at the execution of a single thread.
You need to look at the overall behavior of the application, with the complex object graph spanning multiple threads.
> all you need to do to debug it is to run the program again under the same conditions
this is much easier said than done.
>Other threads may be holding references to the same object
Yes, other threads may be holding references to that object, except when the current thread holds the last one, which was the first thing I said in the statement you're responding to.
But that aside, you seem to be making the assumption that threads sharing state and object graphs between each other is a given, but that's completely false. Reference counting lets you design your application such that threads don't need to interact with each other to manage their memory. So you have the option of completely deterministic behavior, if your problem is amenable to such a solution. On the other hand, tracing GC doesn't have that option. Threads affecting each other is mandatory, even when no object is reachable from two threads simultaneously.
Between reference counting and tracing GC, only one permits deterministic object destruction.
>this is much easier said than done.
You're right, it's much better when your memory management system makes it completely impossible to debug stalls. Completely impossible is definitely better than difficult.
Yes, other threads may be holding references to that object, except when the current thread holds the last one, which was the first thing I said in the statement you're responding to.
But that aside, you seem to be making the assumption that threads sharing state and object graphs between each other is a given, but that's completely false. Reference counting lets you design your application such that threads don't need to interact with each other to manage their memory. So you have the option of completely deterministic behavior, if your problem is amenable to such a solution. On the other hand, tracing GC doesn't have that option. Threads affecting each other is mandatory, even when no object is reachable from two threads simultaneously.
Between reference counting and tracing GC, only one permits deterministic object destruction.
>this is much easier said than done.
You're right, it's much better when your memory management system makes it completely impossible to debug stalls. Completely impossible is definitely better than difficult.
> Computers are finite automata, they can't beget true randomness
Plenty of peripheral inputs that can be used to induce true randomness.
Plenty of peripheral inputs that can be used to induce true randomness.
...of which the computer is not the source.
Our object graph is for a DAW, one of the most complex applications floating around. Multiple threads, some with RT constraints, multiple UIs acting as both view and controller. No GC acceptable here.
as long as you can control when GC runs, it can be acceptable for applications with soft-realtime constraints (not all GC languages give you that level of flexibility though). at least I would prefer that situation over being faced with cascading destruction of refcounted objective-c objects at the most inconvenient moment, with no other recourse than complete rearchitecting.
the GC doesn't solve the problem of no-use-after-destruction, only the timing (and potentially thread) of destructor calling. you still need a refcnt'ing mechanism to make sure that nothing can be considered for GC before nobody is holding a reference to it anymore.
You can absolutely write the same programs without std::move(), using out parameters for objects of expensive construction. Move semantics were introduced because they're a clear ergonomic win.
Gotcha. Probably the best explanation I've seen, thanks for that.
std::move is not only about performance. It is about being able to disallow destructive copies of certain types. I.e. unique_ptr can not be copied, and auto_ptr can. Can you write perfectly functioning code without these features? Sure! You just have to be accurate.
Same with performance. Where nowadays simple std::move() on a return value suffices you had previously explicitly to pass the object/vector with which you swapped as a parameter.
Life with move semantics is a tad easier.
Same with performance. Where nowadays simple std::move() on a return value suffices you had previously explicitly to pass the object/vector with which you swapped as a parameter.
Life with move semantics is a tad easier.
Note that you don't even have to std::move a return value in almost all cases. (The very much non-language-lawyer explanation is that local variables are automatically moved from when you return them.)
Also, if you write
> return std::move(ret);
it can interfere with return value optimisation (RVO). There might be cases where the move makes sense but to my understanding it's enough to just use return.
It's very much possible to take advantage of modern C++ move semantics without an explicit call to move. Maybe you return a unique_ptr created by make_unique in a factory method, or call swap() on a moveable object, etc. I rarely need to write std::move to take advantage of modern C++ although it took some self-education about rvalues etc to learn how to do it.
> return std::move(ret);
it can interfere with return value optimisation (RVO). There might be cases where the move makes sense but to my understanding it's enough to just use return.
It's very much possible to take advantage of modern C++ move semantics without an explicit call to move. Maybe you return a unique_ptr created by make_unique in a factory method, or call swap() on a moveable object, etc. I rarely need to write std::move to take advantage of modern C++ although it took some self-education about rvalues etc to learn how to do it.
Yep, you don't, as long as you want to return a local object. I know that copy elision for returns is not a permitted optimization anymory but a mandatory behavior (since C++11?). And it is a common pitfall to use unnecessary std::move calls which inhibit optimizations.
But if you have something like a builder object which builds a thing which it keeps as an instance variable; then you have to use `return std::move()` or an out-parameter-and-swap if you want to avoid copies.
But if you have something like a builder object which builds a thing which it keeps as an instance variable; then you have to use `return std::move()` or an out-parameter-and-swap if you want to avoid copies.
As u/dryanau says returning std::move breaks the return value optimization such that clang at least will warn about it by default.
Your comment strikes at a deeper issue than what I suspect most replies it gets will address. The example of your never having needed nor missed std::move in all the code you've written rests on the distinction between specific uses and generality.
The reason modern C++ has become what it has become (in both good and bad senses) is because template generic programming has led it down a path of built-in constructs needing to consider an ever-growing zoo of weird cases. For every simple answer to the question, "how should this thing work?" there's a counter-example, "what if the type doesn't have that property?"
That's where we get things like std::move, etc. By contrast when dealing with your own code you can use knowledge of what is actually going to be passed to ignore many of these concerns or change your abstraction-using code to have or not have some property.
Overall, simplifying the use-case or restricting the domain is more pragmatic than making things fully general. C++, in its standard library / template utility headers is painted into a corner in that respect since it doesn't control the use-case, and the effort to restrict the domain (i.e. concepts) was delayed as well as representing itself another layer of complexity rather than a pure simplification.
The reason modern C++ has become what it has become (in both good and bad senses) is because template generic programming has led it down a path of built-in constructs needing to consider an ever-growing zoo of weird cases. For every simple answer to the question, "how should this thing work?" there's a counter-example, "what if the type doesn't have that property?"
That's where we get things like std::move, etc. By contrast when dealing with your own code you can use knowledge of what is actually going to be passed to ignore many of these concerns or change your abstraction-using code to have or not have some property.
Overall, simplifying the use-case or restricting the domain is more pragmatic than making things fully general. C++, in its standard library / template utility headers is painted into a corner in that respect since it doesn't control the use-case, and the effort to restrict the domain (i.e. concepts) was delayed as well as representing itself another layer of complexity rather than a pure simplification.
Well Said.
IMO, a large set of "Modern C++" fanboys have a serious case of "Featuritis"; Merely a focus on the trivial use without understanding the motivation behind it and its place in the larger scheme of building systems. It is like they believe more knowledge of keywords/buzzwords == more expertise.
Even Scott Meyers said this: https://news.ycombinator.com/item?id=27945383
IMO, a large set of "Modern C++" fanboys have a serious case of "Featuritis"; Merely a focus on the trivial use without understanding the motivation behind it and its place in the larger scheme of building systems. It is like they believe more knowledge of keywords/buzzwords == more expertise.
Even Scott Meyers said this: https://news.ycombinator.com/item?id=27945383
> I've been using C++ since 1993. I still don't understand how I've written all this code and never once found myself using, or wishing I could use std::move.
No offense, but are you using modern C++? One of the most common uses of std::move is around ownership transfer of std::unique_ptr - most moderately big projects will have those. Other uses are around preventing copies, etc but those need more thought since you may actually cause regressions by getting in the compiler's way.
No offense, but are you using modern C++? One of the most common uses of std::move is around ownership transfer of std::unique_ptr - most moderately big projects will have those. Other uses are around preventing copies, etc but those need more thought since you may actually cause regressions by getting in the compiler's way.
Well, "modern" is relative. We just recently moved to C++11 :)
We do not use unique_ptr and to be honest I can find little to no use for it anywhere in our 600k lines of C++.
We do not use unique_ptr and to be honest I can find little to no use for it anywhere in our 600k lines of C++.
With apologies to Oscar Wilde: I spent all morning putting in std::unique_ptr and all afternoon taking it out again.
> Even if -Og was ubiquitous, it is still suboptimal to -O0: it can still inline code a bit too aggressively for an effective debugging session.
IMO, the problem is not so much the inlining, but the sad state of the optimization passes losing track of many things and producing useless debug info.
IMO, the problem is not so much the inlining, but the sad state of the optimization passes losing track of many things and producing useless debug info.
It's not just inlining. With -Og (clang or gcc), the debugger frequently can't even tell me the value of function arguments because they've been "optimized out".
While I agree with you and with the OP and I have also been sometimes annoyed by this kind of debugger messages, the more universal solution than wanting an improved debugger is to rely for debugging on some form of progress and error logging in your program, which will work at any optimization level.
Adding such debugging aids in your program may need some extra work, but it also functions in all cases when using the debugger is hopeless, e.g. for concurrent programs or other programs that depend on real-time interactions which prevent the use of breakpoints.
Adding such debugging aids in your program may need some extra work, but it also functions in all cases when using the debugger is hopeless, e.g. for concurrent programs or other programs that depend on real-time interactions which prevent the use of breakpoints.
Yes and no. I use logging too, but that's only useful when I know (or can guess) ahead of time what locations and what things at each location I'll need to know in order to diagnose the problem.
Of course I can always go back and add some logging, but it's not a given that I'll want to keep it around after the problem is fixed. For example if it's in a performance-critical section of code, even the cost of checking to see whether logging is enabled may be too great. And in any case every log statement does add visual noise to the code.
Of course I can always go back and add some logging, but it's not a given that I'll want to keep it around after the problem is fixed. For example if it's in a performance-critical section of code, even the cost of checking to see whether logging is enabled may be too great. And in any case every log statement does add visual noise to the code.
Just want to remind you that `-Og` in Clang is not representative here, as it's exactly the same as `-O1` -- they implemented support for the flag just to ease transitions between GCC and Clang without having to filter out flags, but it doesn't provide any debugging-specific advantage.
Yes, those who think -Og is no good because they are clang users should perhaps file bug reports or contribute improvements instead.
Most game developers have to target Windows, and the main compiler there is MSVC. The other viable alternative is `clang-cl`, which doesn't have `-Og` either.
Bug reports have been filed, especially against MSVC, but they have been largely ignored over the years. AFAIK, MSVC is not open-source so improvements cannot be contributed either.
Bug reports have been filed, especially against MSVC, but they have been largely ignored over the years. AFAIK, MSVC is not open-source so improvements cannot be contributed either.
[deleted]
Am I alone in always debugging in release builds? If you get a crash dump from prod, it'll be a release build. Or if your reproducer takes 2 minutes of (release build!) setup time, you don't want to go through that in debug builds.
Yes, the debugger might have "forgot" the value of that function argument. So check a few other levels of the call stack that it was passed through. Or a local variable might be unavailable in a given line. So have breakpoints a bit before, or trace it into the next function call.
And when all else fails, you can go and look at the disassembly. Yes, calling conventions are stupid, instruction mnemonics are a hassle and register aliases are terrible, but even with rudimentary understanding you can generally figure out what's going on. If your code does a multiplication, look for the `mult`. If your code calls some external function, look for a `call`. Compilers can do cool things with your code, but computers are not magic - you can actually just look at what your CPU does, and it will strongly correlate with your code. As an added bonus, this helps you actually understand what code your compiler produces, which (if you reach for C++!) you probably should care about to some degree.
The only reason I have to care about debug builds is code coverage and/or iterator-checked runs, but that's something that can run overnight if it has to.
/rant
Yes, the debugger might have "forgot" the value of that function argument. So check a few other levels of the call stack that it was passed through. Or a local variable might be unavailable in a given line. So have breakpoints a bit before, or trace it into the next function call.
And when all else fails, you can go and look at the disassembly. Yes, calling conventions are stupid, instruction mnemonics are a hassle and register aliases are terrible, but even with rudimentary understanding you can generally figure out what's going on. If your code does a multiplication, look for the `mult`. If your code calls some external function, look for a `call`. Compilers can do cool things with your code, but computers are not magic - you can actually just look at what your CPU does, and it will strongly correlate with your code. As an added bonus, this helps you actually understand what code your compiler produces, which (if you reach for C++!) you probably should care about to some degree.
The only reason I have to care about debug builds is code coverage and/or iterator-checked runs, but that's something that can run overnight if it has to.
/rant
You're not. Debug builds are rarely that useful and tbh I don't understand why is there such a big fuss about it. The most difficult and most time consuming problems found in production code can only be debugged and, well, understood and hence fixed in production (optimized) code only.
That said, under assumption you have a good code coverage with tests, and your code uses a lot of asserts to check pre- and post-conditions, debug builds can be useful for early detection of broken invariants.
I get that the development cycle is shortened through shorter compile times but that is simply the artifact of all optimizing compilers and not something which is exclusive to C++. And luckily this can be easily alleviated.
Expecting to have comparable runtime performance (which one is it?) in languages backed with optimizing compilers seem a bit delusional to me and even more so if you're in the realm of soft real-time systems.
That said, under assumption you have a good code coverage with tests, and your code uses a lot of asserts to check pre- and post-conditions, debug builds can be useful for early detection of broken invariants.
I get that the development cycle is shortened through shorter compile times but that is simply the artifact of all optimizing compilers and not something which is exclusive to C++. And luckily this can be easily alleviated.
Expecting to have comparable runtime performance (which one is it?) in languages backed with optimizing compilers seem a bit delusional to me and even more so if you're in the realm of soft real-time systems.
I feel your pain. In those cases, you have to try and figure out what the assembly is actually doing and how it relates to the original source code.
Eons ago, i wrote a lot of the first version of debug location tracking/location list tracking in GCC, and the support for evaluating them in GDB (plenty of others came along and improved it nearly instantly - it thankfully did not last long alone ;P).
I have spent, over the years, an amazing amount of time on trying to improve optimized-code debug info.
You could and should improve the optimization passes, but at various points in life they were actually very good. It is always a dance between advancing optimization and debug info, and i certainly will not claim that debug info is thought about as much as it should be - it is often fixed later, as i'm sure you expect.
But at the same time, what the author amounts to ends up reducing to the following:
1. The output executable is pretty well optimized
2. The output debug info is a recreation of the program as-written that can be executed by a debugger
3. The mapping between #2 and #1 has no holes.
IE #2 is basically "you've output the entire program using dwarf5 as a target processor".
This is never going to be a thing that regular users strive for, for lots of reasons. So if the game companies or whoever want this to be the case, they will actually have to contribute more than just bug reports - because it's not something that drives compiler usage for any language, whether it's C++ or anything else (IE it's hubris to believe that other languages won't end up the same way as they get better at optimizing. C++ just has a long head start)
At the same time I am sure that just one good compiler person paid for a year working on any of the compilers to improve optimized debug info would get to a really good place. I know - i was once paid to do it, and when i stopped, it was, back then, in a pretty darn good place compared to where we started[1].
Rather than throwing away every abstraction, it's hard to believe this entire industry can't afford this. Yet I have not seen them try before (perhaps they did after my time, but my time covered 20 years, and it didn't really happen during then)
So it is 100% reasonable to blame game developers or whoever as a group - because as a group, they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them.
[1] I'll note, in practice, the most effective route over time is likely actually to just store enough info to do deoptimize during debugging as-needed, which is the path various VM based languages take and works well. The traditional path to doing this is to JIT the language along the way (which is totally possible for C++), but it's not strictly necessary to do it that way.
The harder part is the debugger/compiler cooperation, not the runtime mechanism.
I have spent, over the years, an amazing amount of time on trying to improve optimized-code debug info.
You could and should improve the optimization passes, but at various points in life they were actually very good. It is always a dance between advancing optimization and debug info, and i certainly will not claim that debug info is thought about as much as it should be - it is often fixed later, as i'm sure you expect.
But at the same time, what the author amounts to ends up reducing to the following:
1. The output executable is pretty well optimized
2. The output debug info is a recreation of the program as-written that can be executed by a debugger
3. The mapping between #2 and #1 has no holes.
IE #2 is basically "you've output the entire program using dwarf5 as a target processor".
This is never going to be a thing that regular users strive for, for lots of reasons. So if the game companies or whoever want this to be the case, they will actually have to contribute more than just bug reports - because it's not something that drives compiler usage for any language, whether it's C++ or anything else (IE it's hubris to believe that other languages won't end up the same way as they get better at optimizing. C++ just has a long head start)
At the same time I am sure that just one good compiler person paid for a year working on any of the compilers to improve optimized debug info would get to a really good place. I know - i was once paid to do it, and when i stopped, it was, back then, in a pretty darn good place compared to where we started[1].
Rather than throwing away every abstraction, it's hard to believe this entire industry can't afford this. Yet I have not seen them try before (perhaps they did after my time, but my time covered 20 years, and it didn't really happen during then)
So it is 100% reasonable to blame game developers or whoever as a group - because as a group, they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them.
[1] I'll note, in practice, the most effective route over time is likely actually to just store enough info to do deoptimize during debugging as-needed, which is the path various VM based languages take and works well. The traditional path to doing this is to JIT the language along the way (which is totally possible for C++), but it's not strictly necessary to do it that way.
The harder part is the debugger/compiler cooperation, not the runtime mechanism.
[gcc/gdb/LLVM outsider here]
If someone is interested in what's being done to improve LLVMs debuginfo, here's a talk by Djordje Todorovic from two years ago: https://www.youtube.com/watch?v=GpMLt1oecOk. It's about tracking dwarf locations for variables by keeping around information from the parent frame. According to that presentation, at least that work has been progressing.
If someone is interested in what's being done to improve LLVMs debuginfo, here's a talk by Djordje Todorovic from two years ago: https://www.youtube.com/watch?v=GpMLt1oecOk. It's about tracking dwarf locations for variables by keeping around information from the parent frame. According to that presentation, at least that work has been progressing.
As for contributions from the gamedev industry, here's a EuroLLVM 2022 presentation from Orlando Cazalet-Hyams working at Sony: https://www.snsystems.com/technology/tech-blog/improving-deb...
The C/C++ ecosystem has really been dragging down the popular expectation of what debugging (and at what level) is possible. They are uniquely bad in breaking programmers' ability to understand what their programs do, with debugging being an essential part of that.
JVMs move heaven and Earth to make optimization not observable and support the JVMTI, upon which debuggers are built. They are required to materialize a bytecode-level view of execution state upon demand, just as CPUs are required to materialize a machine-level view of execution state. As a result, any source debugger written against what is available at the bytecode level will work without needing to see through the abstractions.
JVMs move heaven and Earth to make optimization not observable and support the JVMTI, upon which debuggers are built. They are required to materialize a bytecode-level view of execution state upon demand, just as CPUs are required to materialize a machine-level view of execution state. As a result, any source debugger written against what is available at the bytecode level will work without needing to see through the abstractions.
Thanks for doing that work. Even many years later gcc is in a noticeably better place than clang when it comes to how much debug info is preserved by the optimizer. https://robert.ocallahan.org/2018/11/comparing-quality-of-de...
It strikes me that (2) isn't necessarily even what you want. Optimization will change the character of the program, often radically. Should the debugger reflect the source code says it should do or what the optimized code is actually doing?
This is apparent in C++ when you look at the optimized output on Compiler Explorer: it attempts to color assembly based on the source lines, and some source lines get strewn about the resulting assembly.
This is apparent in C++ when you look at the optimized output on Compiler Explorer: it attempts to color assembly based on the source lines, and some source lines get strewn about the resulting assembly.
> they aren't (again, AFAIK) doing anything about it
Person slightly doing things to LLVM checking in -- the three points you list are compelling, because it's always going to be difficult to describe one program (source) in terms of a vastly different (optimised) program. Do you think there's mileage in showing the developer a partially optimised program [0] instead of trying to perfectly describe the source? It'd be easier to describe in debug-info, and possibly easier for developers to identify unexpected changes to the code they wrote.
[0] "Non-Transparent Debugging of Optimized Code", 1999
Person slightly doing things to LLVM checking in -- the three points you list are compelling, because it's always going to be difficult to describe one program (source) in terms of a vastly different (optimised) program. Do you think there's mileage in showing the developer a partially optimised program [0] instead of trying to perfectly describe the source? It'd be easier to describe in debug-info, and possibly easier for developers to identify unexpected changes to the code they wrote.
[0] "Non-Transparent Debugging of Optimized Code", 1999
> they aren't (again, AFAIK) doing anything about it other than writing blog posts about how the language is mean to them, and filing bugs that compilers are mean to them.
There are people from that industry or adjacent industries writing other languages. Most obviously Jai (Jonathan Blow's language, Jonathan did "Braid" and "The Witness") but you could also argue for Ginger Bill's Odin. I suppose in this context it will be interesting to see whether the debug story is any better in these languages (neither is yet finished).
There are people from that industry or adjacent industries writing other languages. Most obviously Jai (Jonathan Blow's language, Jonathan did "Braid" and "The Witness") but you could also argue for Ginger Bill's Odin. I suppose in this context it will be interesting to see whether the debug story is any better in these languages (neither is yet finished).
It’s always nice to understand more about why game developers do the weird things they do. I always buried and run in “RelWithDebInfo”, then if I really need it, I’ll throw `#pragma clang optimize off` around the code that the debugger can’t see into.
Really nice tip to know! I've also struggled a lot with slow debug performance, so this will be very handy when I know there's a bug somewhere in a specific region but can't pinpoint exactly where.
GCC has something similar:
>#pragma GCC optimize ("O3")
https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Function-Specif...
>#pragma GCC optimize ("O3")
https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Function-Specif...
Yeah, I lost lots of time to switching to a debug build and having to rebuild the whole project since my Debug-build ccache wasn’t warm. The `#pragma` version lets me get my work done quickly and easily.
Have any multibillion dollar game (etc.) companies invested any resources into debug build modes in the open source compilers they develop with? Seems like it should be trivial to tie team productivity rates to quality of the debug experience.
For PS3 development it seems that the devs were forced to use a fork of GCC made to support the notorious Cell architecture (some info from here: https://www.copetti.org/writings/consoles/playstation-3/). Not sure about the NDA details but I would believe contributing to the compiler would have been difficult (and would have not really landed in the GCC trunk anyway).
For PS4 and later Sony used an LLVM fork for their toolchain, so now they don't even have the obligation to share their compiler code. Maybe some improvements to LLVM will "naturally" go back to the devkit, but it's not really a good situation for open-source contribution.
Not sure how it went for the XBox lineups though (Probably it used MSVC, since it's from Microsoft?). Since console development is mired up in NDAs there's not that much public info about it.
For PS4 and later Sony used an LLVM fork for their toolchain, so now they don't even have the obligation to share their compiler code. Maybe some improvements to LLVM will "naturally" go back to the devkit, but it's not really a good situation for open-source contribution.
Not sure how it went for the XBox lineups though (Probably it used MSVC, since it's from Microsoft?). Since console development is mired up in NDAs there's not that much public info about it.
Not exactly what you were asking, but RAD tried to build a debugger for Linux^1, but the project died. I’ve heard the people at RAD described as hard core old school programmers, so if they struggled with this it must just be a hard problem
1. http://www.radgametools.com/debug.htm
1. http://www.radgametools.com/debug.htm
>> To put it bluntly, debugging on Linux is just really bad. We believe it is the biggest roadblock to great software for that platform.
That article is ~10 years old, is that still true?
That article is ~10 years old, is that still true?
For me it is. Linux has great debugging tools, my favorite being valgrind, and I have used rr a few times, but when it comes to traditional debugging, still bad.
For me, gdb is a last resort tool, and none of its frontends satisfy me (Qt Creator is the least bad I tried). I generally prefer developing on Linux, but for me, the Visual Studio (not Code) debugger is the one thing that Windows does way better.
For me, gdb is a last resort tool, and none of its frontends satisfy me (Qt Creator is the least bad I tried). I generally prefer developing on Linux, but for me, the Visual Studio (not Code) debugger is the one thing that Windows does way better.
GDB is still the king of debugging on Linux. GDB is still a really bad debugging experience. GDB front-ends are still incredibly buggy and best avoided unless you wanna debug why your debugger just broke.
That proposition seems risky. If the compiler owners don't accept your pull requests, you risk having to maintain a fork with your changes indefinitely.
Also, I suspect like 95%+ of development at game dev studios happens in MSVC. Nothing to change there.
Unfortunately pestering the compiler devs about this seems like the optimal approach lol
Also, I suspect like 95%+ of development at game dev studios happens in MSVC. Nothing to change there.
Unfortunately pestering the compiler devs about this seems like the optimal approach lol
[deleted]
Maybe? If each AAA studio was putting 6-7 figures into the relevant projects, their change requests should be especially well explained and well formed. A lot of ignored PRs aren't well communicated.
Also, it's kind of moot. For platforms that use GCC, Clang, etc., the maintainers are OSS communities, so that's how you influence the product in a healthy direction for your projects. To sit back and let things happen is also an option, but then debugging support might not meet your needs, for instance.
Seems like mobile and wasm based platforms exist and don't generally use MSVC. Folks around here might be surprised how effective $1MM/yr in engineer salaries can be in the relatively under-resourced OSS compiler development space.
Also, it's kind of moot. For platforms that use GCC, Clang, etc., the maintainers are OSS communities, so that's how you influence the product in a healthy direction for your projects. To sit back and let things happen is also an option, but then debugging support might not meet your needs, for instance.
Seems like mobile and wasm based platforms exist and don't generally use MSVC. Folks around here might be surprised how effective $1MM/yr in engineer salaries can be in the relatively under-resourced OSS compiler development space.
For both productivity and practical reasons (the game still has to be somewhat playable), the use of pragmas to disable optimizations for the file/function of interest is generally recommended over a full debug build.
My question was broad enough to encompass non-portable features like these. Do the games studios (etc.) contribute these features? Or is it solo enthusiasts?
If relevant corps started adding the same hint flags in every toolchain, then conversations about the need for standard mechanisms might be interesting. But even if they don't get standardized, at least some common and portable fundamental libraries can support macros to smooth over the differences.
If relevant corps started adding the same hint flags in every toolchain, then conversations about the need for standard mechanisms might be interesting. But even if they don't get standardized, at least some common and portable fundamental libraries can support macros to smooth over the differences.
If you're developing for games consoles, you're probably stuck using a specific revision of the compiler offered by the platform owner, even if it happens to be clang under the hood. In some cases their certification requirements entirely prohibit using other compilers.
So improved debug performance is probably not in the cards unless you change your code to work around compiler issues.
So improved debug performance is probably not in the cards unless you change your code to work around compiler issues.
Possibly due to this, from what I’ve seen, compiler engineers tend to be in-house working on vendor tools, at the big engine companies, or working on script systems.
MSVC has /Ob1 [1] which only inlines specifically marked functions, or functions defined in a header. Shouldn't this inline all those trivial calls yet keep most debug info if you disable all the other optimizations? I haven't experimented with it yet *or what I rather mean is use Debug builds, but enable /Ob1 in them.
[1]: https://learn.microsoft.com/en-us/cpp/build/reference/ob-inl...
[1]: https://learn.microsoft.com/en-us/cpp/build/reference/ob-inl...
With our rust project we actually compile with `01` because the cost of that optimization pass is less than the cost to our tests running 100x slower. There is definitely a need for a 'more optimal' debug mode, and it is one of those things that feels zero cost until it isn't. Similarly, I try to at least consider compile times when writing code these days.
You know, at least in C++ you get some really poorly performing standard containers. Every time I reach for C, it drives me nuts that the best you get is GLib.
And BOTH pale in comparison to newer language standard libraries like Go's.
And BOTH pale in comparison to newer language standard libraries like Go's.
This dovetails with something I was thinking about recently in the context of Reproducible Builds (or the lack thereof). For most of its history Computer Science (at least as practiced by industry) has gone all-in on "functionally equivalent" transformations, and done almost nothing in the space of "output invariant" guarantees.
In my day job we sometimes we need to reproduce the build of a firmware ROM or executable, sometimes decades after the engineer who last built it left. Getting a match is easier (or even possible) for older tech only because of the relative unsophistication of the compilers used - even then it's only reproducible "by accident," and not because the compiler vendor made any guarantees about X language construct reliably produces Y machine code and data layout.
But we need that! For getting accurate baselines. For security verification. And there's no reason in principle we should have to forego updating compilers, IDEs, and OS environments in the indirect hope of not disturbing anything. Those are two separate things: if we had a through-line of higher level language construct -> semantically defined transformation (irregardless of optimization settings) -> machine code, vendors could continue to update their IDEs and compilers while just making sure they still respect the invariants.
C++'s so-called zero-cost abstractions are poor substitute for this: header (library) writers and C++ gurus write code as if they worked like this "guaranteed output transform" I describe, but no compiler actually has to respect it (nevermind that the fine details of what the transformation actually is isn't nailed down) and it differs between Debug and Release build which is particularly bad for game development as TFA makes clear.
In my day job we sometimes we need to reproduce the build of a firmware ROM or executable, sometimes decades after the engineer who last built it left. Getting a match is easier (or even possible) for older tech only because of the relative unsophistication of the compilers used - even then it's only reproducible "by accident," and not because the compiler vendor made any guarantees about X language construct reliably produces Y machine code and data layout.
But we need that! For getting accurate baselines. For security verification. And there's no reason in principle we should have to forego updating compilers, IDEs, and OS environments in the indirect hope of not disturbing anything. Those are two separate things: if we had a through-line of higher level language construct -> semantically defined transformation (irregardless of optimization settings) -> machine code, vendors could continue to update their IDEs and compilers while just making sure they still respect the invariants.
C++'s so-called zero-cost abstractions are poor substitute for this: header (library) writers and C++ gurus write code as if they worked like this "guaranteed output transform" I describe, but no compiler actually has to respect it (nevermind that the fine details of what the transformation actually is isn't nailed down) and it differs between Debug and Release build which is particularly bad for game development as TFA makes clear.
I worked on a CAD/CAM product, debug mode was literally unusable, it was 100-1000 times slower than optimized mode.
The std::move -> call issue has been fixed in clang-15.
Yes, the article mentions.
> Clang 15.x, also motivated by my #53689 issue, also introduced a similar folding pass for the same functions chosen by GCC (plus std::move_if_noexcept, which I assume GCC maintainers forgot about). This one seems to be enabled by default – see a comparison between Clang 14.x and Clang 15.x on Compiler Explorer.
> Clang 15.x, also motivated by my #53689 issue, also introduced a similar folding pass for the same functions chosen by GCC (plus std::move_if_noexcept, which I assume GCC maintainers forgot about). This one seems to be enabled by default – see a comparison between Clang 14.x and Clang 15.x on Compiler Explorer.
The urge to have basic language features not be part of the language, but instead be part of the standard library is responsible for a lot of the less than stellar parts of of C++. Std::move/forward are particularly obvious cases, but you get similar accessing members of tuples, etc.
Wanting to keep things in the standard library rather than the language itself means that you have to compile large amounts of template hell for a wide array of basic things which hurts build time even in debug modes, and then as this article says you end up with no-op operations that become calls. You also can’t easily debug any of this because you end up with absurd layers of template nonsense for what is again basic functionality.
You can compile with -O1, but that then inlines things that aren’t part of the standard library that makes debugging of the actually relevant code annoying (this is what the debug llvm and clang debug builds do), as it inlines your own code and also means that you lose variables all over the place.
Wanting to keep things in the standard library rather than the language itself means that you have to compile large amounts of template hell for a wide array of basic things which hurts build time even in debug modes, and then as this article says you end up with no-op operations that become calls. You also can’t easily debug any of this because you end up with absurd layers of template nonsense for what is again basic functionality.
You can compile with -O1, but that then inlines things that aren’t part of the standard library that makes debugging of the actually relevant code annoying (this is what the debug llvm and clang debug builds do), as it inlines your own code and also means that you lose variables all over the place.
I'm with Kernigham and Pike: the debugger is for getting a stack trace out of core dumps. Printf is for debugging.
Why limit yourself to crusty text when you can have live visualizations of data structures in various formats, graph dx/dy over time of some variable, stop and inspect data on some condition without having to modify code, quickly start execution on a certain line, quickly drill down into struct members, etc.
I mean it does everything a printf() can do - except just better.
I mean it does everything a printf() can do - except just better.
> Why limit yourself to crusty text when you can have live visualizations of data structures in various formats,
Because that's too fruity, real men (tm) don't use debuggers /s
Programmers are so funny, they'll present themselves as rational and logical, but then follow the advice of some old geezer who thinks syntax highlighting is for pussies rather than admitting that their tools suck and need to be better.
I have a particular chip on my shoulder about this whole printf debugging thing: https://twitter.com/nice_byte/status/1465200027672866816
Because that's too fruity, real men (tm) don't use debuggers /s
Programmers are so funny, they'll present themselves as rational and logical, but then follow the advice of some old geezer who thinks syntax highlighting is for pussies rather than admitting that their tools suck and need to be better.
I have a particular chip on my shoulder about this whole printf debugging thing: https://twitter.com/nice_byte/status/1465200027672866816
For a couple months I had no functioning debugger and was left with only Printf's. I do not ever desire to go back to that. It is far too painful. I think my time to find issues likely was 4x+ longer than before.
Some programs simply have too much state, too complex of structures (for performance or memory reasons), or very long runtimes to reasonably debug purely with prints.
Some programs simply have too much state, too complex of structures (for performance or memory reasons), or very long runtimes to reasonably debug purely with prints.
The one thing printf does better is force me to stop and think about why.
I find that when I use a debugger I fall into the trap of mindless looking at variables, stopping to inspect data (sometimes changing a variable), drilling down into struct members... (That is your exact list except I've never used a debugger that can graph dx/dy though I'm sure that applies too) All the while forgetting that the real goal is to figure out why the program is in the bad state i the first place.
If you can avoid that trap all those tools of debuggers are useful. I sometimes do use a debugger for each of the above things, but I make it a point to limit the time I spend in the debugger.
I find that when I use a debugger I fall into the trap of mindless looking at variables, stopping to inspect data (sometimes changing a variable), drilling down into struct members... (That is your exact list except I've never used a debugger that can graph dx/dy though I'm sure that applies too) All the while forgetting that the real goal is to figure out why the program is in the bad state i the first place.
If you can avoid that trap all those tools of debuggers are useful. I sometimes do use a debugger for each of the above things, but I make it a point to limit the time I spend in the debugger.
I’ve personally never been in a bare-metal environment where the debugger is truly reliable. JVM works great, so does V8 and probably most other VM-style environments, but C++? Swift? Rust? I’ve never seen a truly reliable debugging environment that doesn’t just arbitrarily stop working when I try to inspect a symbol, some subset of the time.
If debuggers were reliable, I’d take them over printf-debugging any day. But to this day I’ve never seen one that is.
If debuggers were reliable, I’d take them over printf-debugging any day. But to this day I’ve never seen one that is.
I didn't expect something as silly as this coming out of Kernighams mouth
You should read "the practice of programming", he (well mostly Rob Pike if I understand the relationship) has some well reasoned criticism of the ways people use debuggers. In fact it is not silly - debuggers often make people feel more productive than they really are, so if you use a debugger (which I do once in a while) be careful to be objective about them.
oh where is that quote? that's great!
In the 'practice of programming'. https://www.cs.princeton.edu/~bwk/tpop.webpage/ I last read it some years ago so I can't tell you exactly what they said. I do know that when pointed out I realized I suffer from the same sins they hate about debuggers: I would step through code mindlessly, changing variables and the like, but never stopping to think. While printf is more painful it forces me to think why is the code in that state, which is key to fixing bugs.
i totally agree
It's also annoying for tools -- like debuggers, and uftrace (user space function tracing), which will start showing all these tiny functions that you don't really care about
I prefer to avoid all the operator overloading and smart pointers, etc.
I prefer to avoid all the operator overloading and smart pointers, etc.
MSVC lacks -Og because it instead treats inlining, optimization, and inclusion of debug information as separate, orthogonal choices.
These days I rarely build a whole MSVC project as "Debug" without optimization. Instead I enable both optimization and debug information, so I always get call stacks and line-by-line stepping ability. When I end up debugging some file where the optimizations inhibit debuggability, I'll recompile just that compilation unit or library without optimization or inlining.
That said, I agree with the gist of the article that both "zero cost abstraction" and debuggability need to be constantly improved.
These days I rarely build a whole MSVC project as "Debug" without optimization. Instead I enable both optimization and debug information, so I always get call stacks and line-by-line stepping ability. When I end up debugging some file where the optimizations inhibit debuggability, I'll recompile just that compilation unit or library without optimization or inlining.
That said, I agree with the gist of the article that both "zero cost abstraction" and debuggability need to be constantly improved.
I think you misunderstand what -Og is. All of the major compilers treat optimization and inclusion of debug information as separate choices, and generating debug symbols for optimized release builds is a very normal thing to do. GCC's -Og is just an attempt at selecting a set of optimizations which don't cause too many problems for debuggability.
It's worth noting that it does a pretty bad job at it. I've had to go back and remove -Og any time I've used it because it just makes so many variables "optimised out".
The author needs to learn about the -Og switch, which can be used together with debugging. It can be thought of as "do those optimizations that don't interfere with debugging". I verified with Godbolt that this suffices to eliminate the call overhead.
Did you even read the article? I mentioned '-Og' several times and its shortcomings.
I read it, but too quickly. Sorry. As you say, the main problem is that only gcc has a usable implementation; the other issues, for me, are an acceptable tradeoff (reasonable speed, but sometimes difficulty because inlining hid something).
I wondered what code I had written that caused such a massive performance hit in debug mode. So I went to profile my code in a debug build to find out. Lo and behold, something like 90% of the CPU time was wasted doing lookups from std::unordered_map because of debug iterators. I tried everything I could to just turn debug iterators off, and eventually gave up and switched to robin_hood::unordered_map which runs great in debug and release. Now I have an unwarranted aversion to the std lib, even though it really is great so long as you're running it in release mode.