My xpeng g9 goes about 570km in summer. Less in winter, like 480 maybe. Longest range ICE i had was a mercedes wagon that went 1050km on one tank of gas.
Filling the wagon today would cost me like 170 euro. Filling my xpeng happens overnight and is about 7-9 euro depending on grid pricing.
At 40gb and a rumoured 5 to 7 TB size of the proprietary flagships you are looking at several megawatts to run one single model instance. Cerebras is insanely power hungry. It is funny how they are essentially a parallell happenstance (chips being made for other compute stuff also works for LLMs) to gaming processors accidentally being good for LLMs.
The world will be much more interesting when real bespoke hardware built for actual LLM usage comes to market. This means silicon of the SIMD flavour or other variants, but using DRAM so you can pack more tightly.
Filler pr jippo fluffer article aside, anyone tried to self host ubicloud lately? A year and a half ago it was super cumbersome, wondering if I should give it a new try now.
This is only a problem if an agent is made in a lazy way (all of them).
Chat completion sends the full prompt history on every call.
I am working on my own coding agent and seeing massive improvements by rewriting history using either a smaller model or a freestanding call to the main one.