We’ve been exploring one specific cost issue in AI products: a lot of async-friendly LLM workloads still run synchronously, which seems to create unnecessary token spend.
I’m curious how people here are handling this in practice for evals, extraction pipelines, classification jobs, or other multi-step workflows.
Are you using batch APIs already? Building internal tooling? Or just accepting the extra cost because batch workflows are too painful to adopt?
We’ve been building an open-source library called ParaLLeM to make it easier to move agent workflows from sync to batch without rewriting everything, and I’d love to understand how others are approaching this problem.
I’m curious how people here are handling this in practice for evals, extraction pipelines, classification jobs, or other multi-step workflows.
Are you using batch APIs already? Building internal tooling? Or just accepting the extra cost because batch workflows are too painful to adopt?
We’ve been building an open-source library called ParaLLeM to make it easier to move agent workflows from sync to batch without rewriting everything, and I’d love to understand how others are approaching this problem.
Repo: https://github.com/parallem-ai/parallem