Ask HN: Why can't Nvidia compete in the inference market?
Hey HN!
Lately I have been thinking about how the inference market will involve. We saw that llama 3 with groq was showing great results. Once question I had was why can't nvidia compete in the inference market? In the video below chamath was saying how nvida won't be able to compete in inference.
https://youtu.be/1ZQ33OnGFWE?t=1254
is here just chilling his bag or are there some truth to this? What are fundamental reason that this would be the case?
2 comments
Likely because they don't make inference accelerators, they make GPUs. It's entirely plausible that someone could optimize the hardware pipeline better by incorporating less complex logic; I wouldn't expect Nvidia to try that though, not with their main products.
This reminds me of a sentence Jensen said internally: "We do the thing that only we can do"