DLRover makes the distributed training of large AI models easy, stable, fast and green
DLRover can restore the training when the process fails without stopping the training job.
In addition to fault tolerance, DLRover provides the flash checkpoint to save/load checkpoint in seconds.
I've personally never trained a model large enough to warrant the use of tools like DLRover, but I definetly see the intended usecase. I do however wonder if re-scheduling a task that failed due to OOM (one of the provided examples) won't just fail again due to OOM on another node.
While you can be as jaded as you want, it's always worth checking if your oppinion has any merit before posting it on an online forum for the rest of the internet to read.
Plastic recycling is a thing. It's more difficult than paper and metal (for instance), due to degradation of polymers and the difficulty of seperating different polymer types from one-another. It's less widespread than some companies would like you to believe (how many times haven't you read "this product was made by X% recycled plastic"?). Most plastic still ends up burned as fossile fuel substitute due to a lack of cost-effective recycling programs, but that does not mean the programs does not exist.
One success story is PET, which is found in drinking bottles. Polymer degration in PET can be repaired and countires with a PET recycling program usually seperates that plastic from other sources before entering a waste compound (such as through designated waste bins). When PET polymers are too damaged to make new bottles they are instead downcycled into synthetic fibers (similar to how degraded paper fibers are downcycled to toilet paper). If you want to read more on the process, why not have a look at Wikipedia?
- https://en.wikipedia.org/wiki/Plastic_recycling