d4l3k · HackerTrans

We're working on making these composable. torchft is largely focused on the model integration and algorithms where as Monarch is handling more of the orchestration/monitoring. They operate at a bit of a different layer but the plan is to have torchft have the fault tolerant algorithms that can be used both in Monarch or a standard PTD job

d4l3k

작년discuss

We want to be tolerant to application bugs and host/GPU failures that can be solved by replacing/restarting the machine. External services and network failures we don't have much control over so aren't aiming to solve that.

For specific types of failures check out the section on "Reliability and Operational Challenges" from the Llama 3 paper https://ai.meta.com/research/publications/the-llama-3-herd-o...

d4l3k

작년discuss

Let me know how it goes! If you're interested in chatting / run into any problems feel free to reach out via the links in my profile

d4l3k

작년discuss

Recently there's been a lot of interest and improvements in semi-synchronous training. The Streaming DiLoCo paper came out this year and is a big step forward for datacenter semi-sync.

Historically it's been limited to areas like federated learning for low power/low network training but with the massive increase in number of GPUs it's becoming relevant even for training in datacenters.

It is another variable ML researchers have to tune so does add some complexity and I expect most folks just aren't familiar with it yet.

On "typed language": all of torchft is typed! The coordination/quorum layers are written in Rust w/ GRPC and the front-end is typed Python with Pyre since it has to interact with PyTorch and model code.

d4l3k

작년discuss

Hey Tim, how's it going?

Interested in lending PyTorch some compute? :)

torchft can handle much larger scales but for public multi-day demonstration run this is what we had available. Point of this blog was to demonstrate correctness of the quorum algorithm and recovery with a stock PyTorch stack and not so much peak flops.

Stay tuned though -- planning on doing some much larger demos on B200s!

d4l3k

작년discuss

Hey, nice to see this here!

I'm the primary author so happy to answer any questions you might have!

d4l3k

3년 전discuss

It seems to work just fine with SIP enabled. I just switched and it seems to be a lot better than Amethyst. Amethyst had a lot of issues with focus follows mouse and dropdown dialogs that seems to just work with Yabai

Seems like SIP is only needed for system dialogs etc so has the same limitations as Amethyst

d4l3k

3년 전discuss

Not sure why they aren't loading, seem to be fine now

They're also at https://github.com/d4l3k/fn.lc/tree/master/static%2Fdiy-self...

d4l3k

3년 전discuss

Supercharger auth is between the car and the charger and doesn't require an internet connection. I get billed the normal way via my Tesla account since the VIN is registered

d4l3k

3년 전discuss

There's some functionality loss but it's mostly been mitigated. I have a custom app I wrote since I can't use the stock app.

The one feature I miss is that there's no voice commands since that requires Tesla's servers but at the same time I also haven't been bothered enough to plug in a custom backend

d4l3k

3년 전discuss

I've got a blog if you're interested haha https://fn.lc/post/

I've been hacking on my car and creating my own self driving models

Code is at https://github.com/d4l3k/torchdrive

d4l3k

3년 전discuss

If the map can't talk to Tesla it'll use Google maps directly. I usually don't allow connections to Tesla on my rooted Model 3

d4l3k

3년 전discuss

I just tried to set this up and couldn't. Seems like it's invite only with a waitlist :/

comments