Over the last year, we kept seeing teams in our ZenML community stretch pipeline DAGs to run agents; they executed things like dynamic branching, state passed through artifact-store workarounds, conditional steps, etc. It technically worked, but the abstraction was fighting the whole way.
The core issue was that pipelines assume you know the graph upfront, but agents don’t. They loop, branch on LLM outputs, pause for human/agent input, and fail in expensive ways when you have to restart from scratch.
Kitaru is our attempt to build the missing layer for that: durable execution for Python agents.
It’s not an agent framework, and it’s not just tracing/observability. It’s the layer underneath existing agent code that gives you crash recovery, pause/resume for human/agent or webhook input, and replay from any checkpoint.
We kept onboarding intentionally simple and small - add `@flow` and `@checkpoint` to normal Python functions and run your agent. No graph DSL, no big rewrite. It’s built on top of ZenML’s engine, so you get persisted artifacts, replayability, and the same code can run locally or on your own infrastructure.
We’re still early, and so I would love to get some feedback on the product and idea.
Author here. I work at ZenML, where we maintain the LLMOps Database (https://www.zenml.io/llmops-database) — a collection of production LLM case studies we've been cataloguing for a while now. This post summarises patterns from the latest 400+ entries (database just crossed 1,200 total).
The findings that surprised me most:
- Context engineering has become a distinct discipline — teams are treating the million-token window as a ceiling to stay under, not a feature to exploit
- Software engineering skills matter more than AI expertise for production success (durable execution, distributed systems, infrastructure work)
- The "wait for the next model" strategy keeps not working — teams shipping today are constraining models, not unleashing them
Thanks! Yes one 'next step' that I'd like to do (probably around the work on deployment / inference that I'm turning to now) will be to see just how small I can get the model. Spacy have been pushing this kind of workflow (models in the order of tens of MB) for years and it's nice that there's a bit more attention to it. As you say, ideally I'd want lots of these tiny models that were super specialists at what they do, small in size and speedy in inference time. As I hinted towards the end of the post, however, keeping all that updated starts to get unwieldy at a certain point if you don't set it all up in the right way.
Interesting. I can maybe try finetuning one or two of the so-called 'uncensored' open models and see if that makes a difference. A bit harder to switch out the dataset completely, as that's really what I'm interested in :) I think the general point that finetuning a model for some custom task works is fairly uncontroversial, but if OpenAI's poor performance was on account of these kinds of guardrails it'd be yet another reason someone might want to finetune their own models I guess.
OpenpPipe - https://openpipe.ai/ - is probably the service that most closely resembles what you’re asking for, but I found the evals weren’t really what I wanted — i.e. following my custom evaluation criteria — so you probably will end up having to do that yourself anyway. But for the finetuning, they’re all somewhat the same. Predibase and OpenPipe are two good options for that. Predibase has more base models for you to finetune, but it’s a bit more unwieldy to work with. I wrote about that in a previous post here -- https://mlops.systems/posts/2024-06-17-one-click-finetuning.....
I followed whatever the guidance was for a specific model. Some of the LLM finetuning providers did indeed set the temperature to 0 and I followed that, but others suggested 1. I could probably iterate a bit to see what is best for each model, and I might well do that for the one that I choose as the one I’ll be doubling down on in subsequent iterations / finetunes. Thanks for the suggestion!
Depends a bit where you’re running etc. This works for Modal, e.g., but they’re just using axolotl under the hood so you can just connect to whatever cloud provider of choice you’re using and then run axolotl straight. I did my finetunes across local GPUs, but it would have been just as easy to do it in a cloud environment using the same axolotl config.
I think not. Normally if you get those kinds of errors you wouldn’t get any output at all. In the blog I show that all 724 of the test cases got proper JSON output etc for the queries so I don’t think this was an issue. I think these kinds of topics would have been well covered in the training data, and probably the OSS models would have used similar data so I don’t even think there’s a disparity to be found between proprietary vs OSS models here.
There's a ton of abstraction in axolotl, for sure, but so far I haven't found that it gets in the way. The main competitor in that space seems to be Unsloth, but that only works with a single GPU machine, so didn't fit my purposes. I'll dive into your blogpost. Thanks for posting!
I am a Software Engineer based in London, UK, and the co-creator of Ekko, a self-deployable realtime infrastructure library with in-transit message processing. (https://ekko-realtime.com)
I have multiple years of experience in the Ruby and JavaScript ecosystems and am comfortable working with Go, PostgreSQL, AWS cloud infrastructure and Docker.
Over the last year, we kept seeing teams in our ZenML community stretch pipeline DAGs to run agents; they executed things like dynamic branching, state passed through artifact-store workarounds, conditional steps, etc. It technically worked, but the abstraction was fighting the whole way.
The core issue was that pipelines assume you know the graph upfront, but agents don’t. They loop, branch on LLM outputs, pause for human/agent input, and fail in expensive ways when you have to restart from scratch. Kitaru is our attempt to build the missing layer for that: durable execution for Python agents. It’s not an agent framework, and it’s not just tracing/observability. It’s the layer underneath existing agent code that gives you crash recovery, pause/resume for human/agent or webhook input, and replay from any checkpoint.
We kept onboarding intentionally simple and small - add `@flow` and `@checkpoint` to normal Python functions and run your agent. No graph DSL, no big rewrite. It’s built on top of ZenML’s engine, so you get persisted artifacts, replayability, and the same code can run locally or on your own infrastructure.
We’re still early, and so I would love to get some feedback on the product and idea.
Happy to answer anything technical.