HackerTrans
TopNewTrendsCommentsPastAskShowJobs

emu

no profile record

Submissions

Google: Largest distributed training job for LLMs across 50000 TPU v5e

cloud.google.com
5 points·by emu·3 năm trước·2 comments

comments

emu
·năm ngoái·discuss
https://x.com/jacobaustin132/status/1886844716446007300 is the announcement of this book.
emu
·năm ngoái·discuss
The uncertainty alone is already stopping people from making job offers this year in physics. Will the department of Energy keep funding fundamental physics research, for example? Will the NSF or NASA? No one knows. The rational response under these circumstances is to pull back on hiring.
emu
·2 năm trước·discuss
> Such capability can’t even be contemplated on the TPU side because you basically get a black box.

I'll just leave this here: https://jax.readthedocs.io/en/latest/pallas/index.html
emu
·3 năm trước·discuss
Baratza also sell replacement parts for their grinders on their web site, and provide clear instructions on how to install them. I am delighted to support a manufacturer that builds repairable products.
emu
·3 năm trước·discuss
But the interpretation of the journalist is the issue here. Comparing a cost optimized part (TPU v5e) against a performance optimized part (H100) and deciding that this makes Google "behind" is just incorrect.
emu
·3 năm trước·discuss
No, it is not. That's the sparse fp8 flop number, but you need to ignore sparsity and compare bf16 flops not fp8 flops for the comparison the ancestor post is making.
emu
·3 năm trước·discuss
Yes, we made this more formally supported recently.

We felt that Windows CPU support was important so everyone can run JAX, even if it's not always the most-accelerated version of JAX. And we got some great PRs from the community that helped fix a few open issues.
emu
·3 năm trước·discuss
We ship Windows CPU only at the moment.

We don't support Windows GPU because we haven't had the engineer bandwidth to support it well.

We recommend WSL2 for GPU on Windows at the moment because that is a compromise: it allows CUDA support, without us having to support another release variant.

But we welcome community contributions!
emu
·3 năm trước·discuss
Not true!

We release Windows CPU wheels (https://pypi.org/project/jaxlib/#files). So JAX on CPU works great on Windows.

We don't release Windows GPU wheels at the moment, but that's because we're a small team and none of us use Windows personally. We welcome contributions!

(I verified that the Windows CUDA GPU support built as recently as two weeks ago, but I don't have the ability to test that it works.)

We recommend WSL2 because that's just using our existing Linux CUDA release.