Debug Mode for LLMs in vLLora(vllora.dev)
vllora.dev
Debug Mode for LLMs in vLLora
https://vllora.dev/blog/debug-mode/
8 comments
Hey, I’m one of the builders behind vLLora, so let me clarify the reasoning.
We split the project intentionally: everything embeddable (the Rust crate you ship inside your own product) is released separately under Apache 2.0. So if you’re embedding it, you’re not inheriting license-key / “licensing baggage” concerns in your codebase. (https://crates.io/crates/vllora_llm)
The parts under the fair-code license are the local debugging UI/tooling. Will always be free to use, we just don’t want it copied and resold.
Any paid, advanced observability lives in a separate cloud offering under a different name so there is no confusion whatsover.
We use it to build deeper agentic workflows. it’s been extremely useful for iterating and we want to share this free to use with everyone. Happy to share our experiences if you want to know more.
Re: "no moat, just a few prompts + Claude/Codex". I’ll be a bit cheeky you’re entitled to that view, but we’re in different camps. Some folks vibe code everything; We believe in having proper tools. You still want a screwdriver for screws.
We split the project intentionally: everything embeddable (the Rust crate you ship inside your own product) is released separately under Apache 2.0. So if you’re embedding it, you’re not inheriting license-key / “licensing baggage” concerns in your codebase. (https://crates.io/crates/vllora_llm)
The parts under the fair-code license are the local debugging UI/tooling. Will always be free to use, we just don’t want it copied and resold.
Any paid, advanced observability lives in a separate cloud offering under a different name so there is no confusion whatsover.
We use it to build deeper agentic workflows. it’s been extremely useful for iterating and we want to share this free to use with everyone. Happy to share our experiences if you want to know more.
Re: "no moat, just a few prompts + Claude/Codex". I’ll be a bit cheeky you’re entitled to that view, but we’re in different camps. Some folks vibe code everything; We believe in having proper tools. You still want a screwdriver for screws.
What a strange naming choice, mixing two things (vLLM and LoRA) while being related to neither..
Haha. One of our objectives is to allow for local debuggging but not just pure debugging; Also enabling users to fine tune a version of the model that performs better. We are working on that feature set and involves Lora :) Hence the name. I guess its a future vision ? :)
[deleted]
interesting but ... why not debug the actual code that is invoking the API.. like break point at the right place, edit state, step over, resume... it seems that the toolchain is a lot more mature and it will fit right into the specific programming environment that is targeted
Because this is way easier. It's effectively a printf debugger and editor you can just slot in the middle of the data stream.
You can still use normal debuggers for the code path, but we found it really valuable to isolate and inspect the agent data stream itself: the exact prompts, model outputs, tool inputs/outputs, and how that impacts cost, time, and behavior over long runs. That visibility has been a big lever for improving overall product quality for some of the deeper agentic experiences we are building. Ability to modify and change models has been useful too.
IMHO the "don't remove anything with a licensekey ever" part in the license is the kind of potential poison that I would never recommend this to my or any other company. More than a few fellow engineers consider nagware an insult and see the potential to twist your arm late in the game making former free functions part of a new "optimized pay package", which you need because you can't fix the bug in the goddamn license part that is a security risk. LOL. (Not saying that you ever do. See below)
And there is no moat, debugging AI flows is a few prompts and a claude code max, google gemini pro or codex whatever for a couple of days while doing the usual things will happen.
Note: Its not about this software specific. I learned that the cuts and bruises of incidents before you come along are the ones that shape behaviour of your partners/colleagues/peers. You may have the purest intentions and best approaches, but someone longe before you ruined it. Its not you, its you chosing the same path.