right now, I am scoping it to nearby people; within X miles distance. Saying strangers look fishy - might have to add more verifications.

How open would you be to trying go out?

essekar

·2 bulan yang lalu·discuss

anyone has any videos?

essekar

·3 bulan yang lalu·discuss

lol, that is what we all needed. AI Boss

essekar

·3 bulan yang lalu·discuss

anything tbh. as long as you have runbook - you can try to automate actions through nvsx; it sits on top of NVSentinel. restarting will work mostly for smaller jobs - distributed training, pretty common will need more fault tolerant methods to continue rather than just restarting.

essekar

·3 bulan yang lalu·discuss

[dead]

essekar

·4 bulan yang lalu·discuss

One of the feedbacks we got while testing was - we might even reduce the duration of checkpointing. Which was a huge insight.

We are testing out & would love more collaboration from the community - if your team is running training jobs, hit us up.