Understand, we'll definitely think about the network part. Just in case it may help, if `nvidia-smi nvlink -gt d` is useful for you in this context then there is a related metric NVLink Throughput Rate to compare runs and monitor. At least you might get an idea whether/how internal links are utilized.
For now it only tries to extract NCCL time percentage from the profile, if available, and show it profile summary. Some hints count be in the step trace timeline as well. We are planning to record some NCCL related counters separately as well.
Thanks for the feedback, we'll be working on it for sure. At least an explanatory screencast is in the works now, other info material, use cases, etc. are planned.
Many training and inference workloads run in the cloud or on remote servers and profiling them is not straightforward. Having a SaaS makes things much simpler, and also enables additional features, such as team access and sharing. As far as the data privacy is concerned, profiles do not contain any model or raw data, just resource usage, execution statistics, etc., which is acceptable for most of the users to send to a third party. And for the business model, in my opinion, SaaS allows to better monetize the offering and ensure better and up-to-date end product in this case. But this is open, we may consider a free client-side version as well at some point.
Since ML monitoring is rather a broad term that can be applied to model development, evaluation, retraining and production stages, I'd like to give more context on what Graphsignal is designed for. Our focus is the operational aspects of models deployed to production, e.g. incoming data validity, sudden drift in input and/or output data, etc. making it possible to troubleshoot issues when they detected. So it designed to help MLOps, DevOps and SRE teams to ensure production models' performance and availability.
Good point, thanks! The idea behind these benchmarks is to make the results usable in real-world programs, rather than benchmarking real-world programs. I rephrased that sentence to avoid any confusion.
We haven't tested it with celery yet. It looks like it should work. gevent is supported by blocking call profiler, and CPU and memory profilers as well as exception and metric reporting are library independent.
We haven't tested the whole agent with asyncio applications yet. I guess only CPU profiler was tested during development. We'll do and include it in the docs. For now, if you see any problems, please just open a ticket. Thanks!
Yes, the apps were under simulated CPU load, memory allocations, etc. The good thing with sampling profilers is that overhead stays relatively stable even under high load.
We are measuring both, individual profiler overhead when active (printed by the agent in debug mode) and total CPU and memory overhead of the app running over long periods of time with and without agent.
Current agent and the dashboard are designed for long-running applications, such as servers or scripts. There are no plans for end user devices yet. But because the agent is pure Python (it just relies on some system specific functionality, such as signalling), it could work with a few tweaks.
StackImpact is a set of profilers, which continuously sample production applications at low-overhead. The result is line-of-code precision, not just application-level metrics. I think it doesn't really compare to monitoring tools such as DataDog. However, it also sends metrics as well (cpu, memory, GC). (Disclaimer: I work at StackImpact)
There is no on-prem offering yet, since there was actually no demand/requests. At least with the Golang agent, which was introduced first. With Python agent we will reprioritise it. Thank you for the feedback! (Disclaimer: I work at StackImpact)