Every function would be pretty overkill, but you can automatically install instrumentation on functions on demand or with a pre-determined user selection. You can even instrument on a line basis if your instrumentation is cheap enough. I've experimented with this in a VS extension I'm developing and I could easily browse through a non-trivial game codebase without causing noticeable performance overhead [1]. In the demo, the instrumentation is auto-installed on all functions within the file you opened. Obviously, this is just one project I was testing on but it shows that this type of tracing is feasible.
I sympathize a lot with this post. Today, modern IDE's still don't provide automatic ways to inspect program execution and nicely visualize it. Instead, they rely on having you place all the breakpoints, step-step-step and restart everything again if you did not inspect something in time. I don't want to insert print statements that end up clogging the console completely. This workflow also quickly breaks with a lot of logging.
I'm quite a bit into developing more tooling around that, and it's really hard - developers particularly want tools that completely integrate with existing IDE's. Debugger infrastructure is usually not built to be very exstensible or accessible. For my extension for Visual Studio https://d-0.dev/ I had to put serious effort into making sure it all still works correctly with VS breakpoints which is extremely difficult when the debugger expects all of the code to be in the same place all the time.
I'm not sure about your particular case, but if your environment really is headless, then it should absolutely be possible to run it a lot faster than realtime. It depends on what the environment is and if you have access to its source code (we do not have that in TrackMania so it's a lot harder). Either the environment is purposely throttling the amount of time it simulates, or it just takes so much time to simulate the environment that it's not possible to speed it up anymore.
We're lucky in case of TrackMania because it internally has systems to both set the relative game speed and also completely disable all rendering and just run physics. Linesight achieves about ~10x speedup where the most time spent now is in rendering game frames and running the inference on the network. They also parallelize training by running more game instances and implementing a training queue. For the "raw" speedup ratios, TM usually achieves about ~60x (one minute is simulated in one second) and I use this speedup to implement bruteforce functionality in the tool (coupled with a custom save states implementation).
This is an awesome overview and if you want more, most of those are documented in an approachable way on YouTube.
Just wanted to provide some perspective here on how many things those projects need to take care of in order to get some training setup going.
I'm the developer behind TMInterface [1] mentioned in this post, which is a TAS tool for the older TrackMania game (Nations Forever). For Linesight (last project in this post), I recently ended up working with its developers to provide them the APIs they need to access from the game. There's a lot of things RL projects usually want to do: speed up the game (one of the most important), deterministically control the vehicle, get simulation information, navigate menus, skip cut scenes, make save states, capture screenshots etc. Having each of those things implemented natively greatly impacts the stability and performance of training/inference in a RL agent, e.g. for the latest version the project uses a direct capture of the surface that's rendered to the game window, instead of using an external Python library (DxCam). This is faster, doesn't require any additional setup and also allows for training even if the game window is completely occluded by other windows.
There are also many other smaller annoying things: many games throttle FPS if the window is unfocused which is also the case here, and the tool patches out this behaviour for the project, and there's a lot more things like this. The newest release of Linesight V3 [2] can reliably approach world records and it's being trained & experimented with by quite a few people. The developers made it easy to setup and documented a lot of the process [3].
Hello, since some time now I develop an extension for Visual Studio that makes it possible to see what's going on in your program live. This includes which code paths are executed and what changes in variables take place on each line of the function. Here, I decided to create a prototype that makes it possible to visualize the longest executing lines in the function, also completely live and at runtime.
The difference between this and e.g profilers like Tracy is that there's no need to insert the instrumentation manually, and you can explore the codebase much more freely to see what's going on. This is done through dynamic function relocation and inserting an rdtsc call after each line to collect the CPU timestamps. There's some more supporting code of course, but when it comes to overhead we get about ~15 instructions per line (more if it supported multiple threads).
On top of all that, the extension works really hard not to disrupt any standard Visual Studio workflow like setting breakpoints or viewing the call stack, which has been a very hard thing to achieve: the VS debugger expects that all code is on the same place as it was and that it maps perfectly to the corresponding debug information (which is not at all the case when the extension is running!).
There's also a lot of apps written in Vala for elementary OS's AppCenter with screenshots & links to GitHub: https://appcenter.elementary.io/ (not all, but most). I published two apps there and using Vala has been a great experience for creating Gtk/Linux desktop apps.
I'll add to that my own Visual Studio extension that improves runtime debugging by showing you which lines are actively executing and what changes happened line-by-line: https://d-0.dev It integrates with the VS debugger completely and does not require any changes to one's codebase.
It's also available on the marketplace if you want to try it out: https://marketplace.visualstudio.com/items?itemName=donadigo...
Hello HN, for about 6 months I've been developing a Visual Studio extension that makes it easier to debug & inspect dynamic C++ code that changes state e.g every frame. The newest feature I added to the extension is showing real-time changes alongside each line. This works kind of like a global, immediate watch, but without any breakpoints - this is really useful in game development where you'd like to track variables that change state all the time.
The feature works by setting the cursor inside a function, and the extension will start showing you the code paths that are executed and what local variables are changing in it. You can also use it as a function recorder with no breakpoints - just set the cursor in the function, execute an action that calls the function and you'll be able to trace everything that the function did.
Interesting approach. Was it enough for your use case despite being quite cumbersome if you end up needing to save a bunch of different more complex instances? I mean, I imagine most dynamic objects that are the real state you'd care about will probably be allocated somewhere on the heap and be referenced from lists or pointer types. I guess this would work best for POD types like program settings and the like, so I'm curious about your setup there.
Hey, looks like we had the same idea! I've run into the same issue and wanted to stop building debug UI's that do the same thing.
In my case, I went to implement this for Windows, but bit more advanced with inline scripting support. Basically, you insert a snippet anywhere in the code that can call various functions and one of them is `view()` which will show all locals (or a chosen object out of scope) in a Visual Studio window (it's a VS extension [0], website [1]). Calling it again will update the view with the new state. The scripting language underneath is Angelscript and it understands symbols inside your entire project (e.g you could do `view(Namespace::g_variable)` or even call functions from the program etc.
The snippet hook is then JIT'ed into the target function by relocating it using info from the PDB. This took me ~1y to implement, and it's still a research phase project.
Hey HN, Adam here. I've been building D0 - an extension that adds non-standard debugging features that help debug C++ applications in real time without setting any breakpoints.
D0 was previously a separate application, now it's exclusively a Visual Studio extension (with integration for other editors coming soon).
With D0 I'm focusing on features that make you less likely to place a breakpoint and more likely to observe what is going on immediately. Currently the extension has 4 main features:
1. Shows live code execution in the Visual Studio editor as it happens. This eliminates the need for setting breakpoints when you just need to see if a line is running or not which happens surprisingly quite often. And if you do need to inspect the values with the application in break mode, the line indicators give you a much better idea of e.g. when to expect that a breakpoint will be hit.
2. Shows live call stack of the current function. There's often times a need to inspect the call stack of a particular function. Usually this requires putting a breakpoint and inspecting the call stack while the application is in break mode. D0 improves on this and shows all paths that a function was called with previously just by putting the editor cursor anywhere in that function. This is much more ergonomic if you need to iterate all callers as you do not need to continuously break the application and remember who called the function.
3. Allows inserting scripting snippets anywhere in the code. These snippets can print or view complex objects without recompiling the main program and can be easily toggled on/off. This enables a very fast iteration on e.g the set of logs you need to produce at runtime since you can just enable/disable certain snippets without any waiting.
4. View objects live - tied with the scripting snippets, you can put a view() call in your main code with a snippet and D0 will create an object view inside Visual Studio that allows you to monitor variables live and change their values. This can be helpful when you need to tweak some settings/properties within the running application and it's basically a runtime version of the Locals window in Visual Studio when you hit a breakpoint.
There's more features that I'm working on and I'm looking forward to your feedback!
To each their own but the notion that the debugger "will not scale to hundreds of processors and terabytes of data" just seems bizarre to me. If a debugger cannot scale to such quantities (which on its own can analyze a lot of data because it's running on the same/another system), then why could I manage to see what is going on? I guess what is meant here is that with sufficient knowledge about the codebase I'm working on, I'll be able to localize the issue much quicker than a debugger, simply because I have an understanding of what to actually look for. But I still don't know why a smart debugger couldn't help me localize the issue.
The one thing I agree about is that debuggers got stuck in a point in time and they cannot do much more than set a breakpoint and trace some code (with exceptions) - but this is not a limitation of the concept, it's rather that the tools we have today often fail to help us. We resort to using printf() instead because then we're looking at just one variable and when it's printed, rather than stopping the whole program or hitting the continue shortcut 50 times.
It's why I've been researching the topic quite a bit myself with some other ideas than "stop the whole thing and look at state". One thing I found is that if the debugger is not integrated into your workflow (it's not working by default after you compile/run), you'll be much less likely to actually use it. Another way to look at it is: would you still not utilize the information the debugger displays, even if it displayed the state at a line you highlighted immediately?
Hey, ever since I saw developer tools in today's web browsers, I wanted to have that kind of functionality available when developing C++ projects. The ability to browse the DOM tree, modify it and also execute JS in the console seemed like something that could be brought into other languages, albeit with differences of course.
Around half a year ago I started work on D0, a Windows application that enables some of that functionality for C/C++ programs compiled using MSVC and Clang. The aim is to enable viewing objects/structs in your code live and also making it possible to call C++ functions completely at runtime, without requiring any modifications to your C++ code. The project is still in early stages of development, but I'm already using it e.g to develop itself.
D0 takes a different approach than most debuggers, in that it integrates a scripting language (Angelscript) into the workflow. The scripting engine then allows for inserting scripting snippets anywhere into your C++ code at runtime in which you can call functions like `view(object);` or `log(object);` to view application state at the point where you inserted the call. Unlike breakpoints, this call becomes a part of your application code and is executed accordingly without stopping the entire application. This allows for viewing/modifying the state live, instead of e.g repeatedly continuing from a breakpoint or building a custom UI for that state.
The tool has been quite useful for me already, but I'm currently looking for early feedback on the idea. If you have any questions or suggestions, please ask!
The article provides a pretty good overview of the landscape of debuggers out there. I find myself using printf debugging whenever something is a dynamically evolving state and I need to monitor just the relevant bits of it and breakpoints when something is either crashing or requiring a deeper thought about how the code executes.
That said, I still think there's quite a bit of improvement to be made, which is why I started building a new debugger for myself which puts a lot more focus on breakpoint-less workflow, speed of iteration and scripting ability: (demo) https://www.youtube.com/watch?v=qJYoqfTfuQk
It's geared mainly for gamedev, but I also do use it many times to e.g debug itself.
I got quite frustrated by how much native C++ programs are slow/closed to any kind of introspection/debugging so this year I will be attempting to build a live object inspector/debugger for C++ programs/games, independent of the game engine you are using. I have currently lots of ideas on features I want to build but at least for me it's an interesting challenge to tackle. The basic idea right now is to allow to add small changes to your code live without recompiling the whole thing and I aim for instantaneous compilation times, so adding a print() does not take >=5 seconds to compile. Currently I achieve this by using a scripting language that is exposed to classes/functions/variables in your codebase but I may change the approach later if I find a better solution.
I currently have a prototype but nothing that could be used in production yet: https://donadigo.com/d0/
[1] https://www.youtube.com/watch?v=3PnVG49SFmU