We're scheduler agnostic and works well with autoscaling. We have tested against the k8s default scheduler and Volcano. Tests for Kueue and Yunikorn are coming shortly.
Hi all — I’m Dan, founding team at clockwork.io. Today we launched TorchPass! We'd love to get your feedback.
tl;dr: we built TorchPass because large distributed training jobs fail a lot, and checkpoint restart is expensive. TorchPass addresses this by migrating the state from failed resources to spares.
In large GPU clusters, even small failures (a GPU falling off the bus, a node crash, a network link flap) can bring down an entire distributed training job. And once you get into clusters with hundreds or thousands of GPUs, something is almost always failing. Research from Meta suggests mean time to failure drops to about 7.9 hours for a 1,024-GPU cluster. And when a single failure occurs, the entire distributed job crashes.
The usual recovery model is to take frequent checkpoints during training, and recover from the most recent checkpoint when a failure occurs. But:
all work since the last checkpoint is lost
time is wasted replacing nodes and checkpoint reloading
more time is lost restarting the entire distributed job
TorchPass uses a different approach: instead of restarting the job, it migrates the failed training rank to a spare GPU and resumes training at the same step.
TorchPass supports planned migration (triggered pre-emptively when an imminent failure is detected) or unplanned migration (triggered by a hard failure). Further details about how it works can be found here:https://clockwork.io/blog/torchpass-workload-fault-tolerance...
We ran a 3,000 step training benchmark using TorchTitan Llama-4 MoE Scout (109B) on 64 H200 GPUs with random failure injection to compare checkpoint restarts, TorchPass and TorchFT.
TorchPass completed in 405 min
Checkpoint restart completed in 818 min.
TorchFT in 930 min
Checkpoint restart was slower mainly because of the time taken to restore from checkpoint, restart the training and recompute the work since the last checkpoint.
TorchFT lost almost no time due to the failures, but was slower because it introduced a significant per-step overhead because it requires the using gloo (rather than NCCL) for cross replica all reduce operations.
Happy to answer questions about the implementation and benchmarks.
One thing missing in the blogpost is in practice you see many large orgs, especially in finance, living with multiple time domains. For example, on-prem trading systems almost always use PTP or PPS for sub-microsecond timestamping, often on dedicated networks to reduce jitter (for meeting regulatory requirements like MiFID II and CAT) while the rest of their infra (in on-prem and cloud) just runs NTP for millisecond-class sync. Both protocols are fundamentally sensitive to network conditions — the mean offset may look fine, but outliers due to congestion/jitter can be very poor.
The consequence of having multiple time domains is pretty painful when you need to reconcile logs or transaction histories across systems with different sync accuracy. Millisecond NTP logs and sub-microsecond PTP logs don’t line up cleanly, so correlating events end-to-end can become guesswork rather than deterministic ordering.
If you want reliable cross-system telemetry and audit trails, you'll need a single, high-accuracy time sync approach across your whole stack.
A very clever part of the HUYGENS algorithm is that it doesn’t just sync clocks pair-wise, it leverages a natural network effect where a group of pair-wise synchronized clocks becomes transitively synchronized, helping reduce errors further without requiring specialized hardware. That’s one of the key reasons it can achieve ~100 nanoseconds of software-based sync on commodity networks.
The authors’ work forms the basis of what the team at Clockwork.io is building, enabling accurate one-way delay measurements (rather than just RTT/2) that improve latency visibility and telemetry across CPU and GPU infrastructure