right now, I am scoping it to nearby people; within X miles distance. Saying strangers look fishy - might have to add more verifications.

How open would you be to trying go out?

essekar

·2 maanden geleden·discuss

anyone has any videos?

essekar

·3 maanden geleden·discuss

lol, that is what we all needed. AI Boss

essekar

·3 maanden geleden·discuss

anything tbh. as long as you have runbook - you can try to automate actions through nvsx; it sits on top of NVSentinel. restarting will work mostly for smaller jobs - distributed training, pretty common will need more fault tolerant methods to continue rather than just restarting.

essekar

·3 maanden geleden·discuss

[dead]

essekar

·4 maanden geleden·discuss

One of the feedbacks we got while testing was - we might even reduce the duration of checkpointing. Which was a huge insight.

We are testing out & would love more collaboration from the community - if your team is running training jobs, hit us up.