With batched parallel requests this scales down further. Even a MacBook M3 on battery power can do inference quickly and efficiently. Large scale training is the power hog.
I was daydreaming of a special LLM setup wherein each token of the vocabulary appears twice. Half the token IDs are reserved for trusted, indisputable sentences (coloured red in the UI), and the other half of the IDs are untrusted.
Effectively system instructions and server-side prompts are red, whereas user input is normal text.
It would have to be trained from scratch on a meticulous corpus which never crosses the line. I wonder if the resulting model would be easier to guide and less susceptible to prompt injection.
The fan increases air speed at the centre of the rotor, creating a low pressure zone which then sucks in surrounding air. So it helps to place the fan away from the window (roughly far enough that the wind cone "fits" the opening).
http://paulcollier.ca/