Ask HN: Browser-Based LLM Models?

4 pointsby lulzury2 anni fa5 comments

Does anyone know if there are there any plans for browsers to natively integrate LLMs, LLM APIs, or LLM models like Llama for local use by web applications?

I feel there's a large opportunity here for a more privacy-friendly, on-device solution that doesn't send the user's data to OpenAI.

Is RAM the current main limitation?

5 comments

throwaway888abc2 anni fa

https://simonwillison.net/2024/Jul/3/chrome-prompt-playgroun...

https://developer.chrome.com/docs/ai/built-in

lulzury2 anni fa

Thank you! This is exactly what I was looking for. I hope these become part of the web platform APIs! With Google pushing this effort, it might be highly likely.

I hope Apple will follow suit with some of their small models (https://huggingface.co/apple/OpenELM).

And then maybe even Firefox will join them...

throwaway4259332 anni fa

Every big tech company is trying to do this. FB (through whatsapp), Google (through chrome/Android), Apple (through Safari/iOS/etc). As soon as they meet their internal metrics, they will release these to public

FrenchDevRemote2 anni fa

"Is RAM the current main limitation?"

(V)RAM+processing power+storage(I mean what kind of average user wants to clog half their hard drive for a subpar model that output 1 token a second?)

Crier10022 anni fa

check out https://github.com/mlc-ai/web-llm

IMO the main limitation is access to powerful GPUs for running models locally and the size of some models causing UX problems with cold starts