Does anyone know how this “user decides how much compute” is implemented architecturally? I assume it’s the same underlying model, so what factor pushes the model to <think> for longer or shorter? Just a prompt-time modification or something else?
I see a couple comments comparing llama.cpp and Ollama, and I think both have utility for different purposes. Having used both llama.cpp (which is fantastic) and Ollama, a couple things that I find valuable about Ollama out-of-the-box --
- Automatically loading/unloading models from memory - just running the Ollama server is a relatively small footprint; every time a particular model is called it is loaded into memory, and then unloaded after 5 mins of no further usage. It makes it very convenient to spin up different models for different use-cases without having to worry about memory management or manually shutting down those tools when not in use.
- OpenAI API compatibility - I run Ollama on a headless machine that has better hardware and connect via SSH port forwarding from my laptop, and with a 1 line change I can reroute any scripts on my laptop from GPT to Llama-3 (or anything else).
Overall, at least for tinkering with multiple local models and building small, personal tools, I've found the utility:maintenance ratio of Ollama to be very positive -- thanks to the team for building something so valuable! :)
I also had a similar thought process regarding understanding vs memorizing facts while transitioning from studying CS (where I emphasized understanding the underlying concepts rather than trying to memorize atomic facts that I could derive) to medicine (where having facts memorized is also a key component). Interestingly, I found that committing to memorizing facts actually helped me gain a deeper understanding of the topics themselves, which was not what I originally expected! (I wrote a little bit about the above a few months ago -- https://samrawal.substack.com/p/on-the-relationship-between-...)