right now, I am scoping it to nearby people; within X miles distance. Saying strangers look fishy - might have to add more verifications.

How open would you be to trying go out?

essekar

·2 mesi fa·discuss

anyone has any videos?

essekar

·3 mesi fa·discuss

lol, that is what we all needed. AI Boss

essekar

·3 mesi fa·discuss

anything tbh. as long as you have runbook - you can try to automate actions through nvsx; it sits on top of NVSentinel. restarting will work mostly for smaller jobs - distributed training, pretty common will need more fault tolerant methods to continue rather than just restarting.

essekar

·3 mesi fa·discuss

[dead]

essekar

·4 mesi fa·discuss

One of the feedbacks we got while testing was - we might even reduce the duration of checkpointing. Which was a huge insight.

We are testing out & would love more collaboration from the community - if your team is running training jobs, hit us up.