Show HN: docker/model-runner – an open-source tool for local LLMs(github.com)
github.com
Show HN: docker/model-runner – an open-source tool for local LLMs
https://github.com/docker/model-runner
10 comments
Hi everyone, we're the maintainers.
We're rebooting the model-runner community and wanted to share what we've been up to and where we're headed.
When we first built this, the idea was simple: make running local models as easy as running containers. You get a consistent interface to download and run models from different backends (llama.cpp being a key one) and can even transport them using familiar OCI registries like Docker Hub.
Recently, we've invested a lot of effort into making it a true community project. A few highlights:
- The project is now a monorepo, making it much easier for new contributors to find their way around.
- We've added Vulkan support to open things up for AMD and other non-NVIDIA GPUs.
- We made sure we have day-0 support for the latest NVIDIA DGX hardware.
We're rebooting the model-runner community and wanted to share what we've been up to and where we're headed.
When we first built this, the idea was simple: make running local models as easy as running containers. You get a consistent interface to download and run models from different backends (llama.cpp being a key one) and can even transport them using familiar OCI registries like Docker Hub.
Recently, we've invested a lot of effort into making it a true community project. A few highlights:
- The project is now a monorepo, making it much easier for new contributors to find their way around.
- We've added Vulkan support to open things up for AMD and other non-NVIDIA GPUs.
- We made sure we have day-0 support for the latest NVIDIA DGX hardware.
Love that it's open source and the addition of Vulkan support.
Docker model run is now part of my demos when deploying ml stack stuff, pretty sure that this is removing the entrypoint of using multiple tools to just do inference, this is great!
Any new features you think we should add to further enhance your usage? Glad you find it useful
Really glad to see DMR getting "new life"... I’ve been experimenting with it for local agentic workloads (MAF, Google's ADK, cagent, Docker MCP, etc...) and it’s such a clean foundation...
A few things that could make it even more powerful (maybe some are out of your scope):
- Persistent model settings (context size, temperature, etc.) across restarts — right now it always resets to 4k, which breaks multi-turn agents. - HTTP/gRPC interface to let tools and frameworks talk to DMR directly, not only through the CLI. (Here the issue is on Docker MCP side, right?) - Simple config management (`docker model set` or `docker model config`) so we can tweak GPU, threads, precision, etc. predictably. (there are at least a couple of issues on this topic already...)
TBH, I love how fast the discussion evolved today.
Congrats and good luck with this. I'll try to help, promised!
A few things that could make it even more powerful (maybe some are out of your scope):
- Persistent model settings (context size, temperature, etc.) across restarts — right now it always resets to 4k, which breaks multi-turn agents. - HTTP/gRPC interface to let tools and frameworks talk to DMR directly, not only through the CLI. (Here the issue is on Docker MCP side, right?) - Simple config management (`docker model set` or `docker model config`) so we can tweak GPU, threads, precision, etc. predictably. (there are at least a couple of issues on this topic already...)
TBH, I love how fast the discussion evolved today.
Congrats and good luck with this. I'll try to help, promised!
Keep opening pull requests and issues, we need these things, you are right!
Nice, I really like the recent Vulkan support.
Thanks very much. It worked well for you? Which hardware? :) Any other feedback, keep it coming!
We're the maintainers of docker/model-runner and wanted to share some major updates we're excited about.
Link: https://github.com/docker/model-runner
We are rebooting the community:
https://www.docker.com/blog/rebooting-model-runner-community...
At its core, model-runner is a simple, backend-agnostic tool for downloading and running local large language models. Think of it as a consistent interface to interact with different model backends. One of our main backends is llama.cpp, and we make it a point to contribute any improvements we make back upstream to their project. It also allows people to transport models via OCI registries like Docker Hub. Docker Hub hosts our curated local AI model collection, packaged as OCI Artifacts and ready to run. You can easily download, share, and upload models on Docker Hub, making it a central hub for both containerized applications and the next wave of generative AI.
We've been working hard on a few things recently:
- Vulkan and AMD Support: We've just merged support for Vulkan, which opens up local inference to a much wider range of GPUs, especially from AMD.
- Contributor Experience: We refactored the project into a monorepo. The main goal was to make the architecture clearer and dramatically lower the barrier for new contributors to get involved and understand the codebase.
- It's Fully Open Source: We know that a project from Docker might raise questions about its openness. To be clear, this is a 100% open-source, Apache 2.0 licensed project. We want to build a community around it and welcome all contributions, from documentation fixes to new model backends.
- DGX Spark day-0 support, we've got it!
Our goal is to grow the community. We'll be here all day to answer any questions you have. We'd love for you to check it out, give us a star if you like it, and let us know what you think.
Thanks!