Sounds good for tasks like the excel example in the article, but I wonder how this approach will hold up in other multi-step agentic flows. Let me explain:
I try to be defensive in agent architectures to make it easy for AI models to recover/fix workflows if something unexpected happens.
If something goes wrong halfway through the code execution of multiple 'tools' using Programmatic Tool Calling, it's significantly more complex for the AI model to fix that code and try again compared to a single tool usage - you're in trouble, especially if APIs/tools are not idempotent.
The sweet spot might be using this as a strategy to complete tasks that are idempotent/retryable (like a database 'transaction') if they fail half way through execution.
We ended up making middleware for LLM 'tools/functions' that take common data/table formats like CSV, Excel and JSON.
The tool uses an LLM to write code to parse the data and conduct the analysis to return back to the LLM. Otherwise, we found pumping raw table data into a LLM is just not reliable, even if you go to the effort to conduct analysis on smaller chunks and merge the results.
I think the best way to explain this is to provide an example.
Scenario: A B2B fintech company processes chargebacks on behalf of merchants, this involves dozens of steps which depend on the type & history of the merchant, dispute cardholder. It also involves collection of evidence from the card holder.
There's a couple of key ways that LLMs make this different from manual workflows:
Firstly, the automation is built from a prompt. This is important as it means people who are non-technical and are not necessarily comfortable with non-code tools to pull data from multiple places into a sequence. This increases the adoption of automations as the effort to build & deploy them is lower. In this example, there was no automation in place despite the people who 'own' this process wanting to automate it. No doubt there's a number of reasons for this, one being they found todays workflow builders too hard to use.
Secondly, the collection of 'evidence' to counter a chargeback can be nuanced, which often requiring back and forth with people to explain what is needed and check the evidence is sufficient against a complicated set of guidelines. I'd say a manual submission form that guides people through evidence collection with hundreds of rules subject to the conditions of the dispute and the merchant could do this, but again, this is hard to build and deploy.
Lastly, LLMs monitors the success of the workflow once it's deployed, to help those who are responsible for it measure its impact and effectiveness.
The end result is that a business has successfully built and deployed an automation that they did not have before.
To answer your second question, dynamic routing describes the process of evaluating how complicated a prompt or task is, and then selecting an LLM that's 'best fit' to process it. For example, short & simple prompts should usually get routed to faster but less intelligent LLMs. This typically makes users happier as they get results more quickly. However, more complex prompts may require larger, slower and more intelligent LLMs and techniques such as 'reasoning'. The result will be slower to produce, but will be likely be far more accurate compared to a faster model. In the above example, a larger LLM with reasoning would probably be used.
At Zenning AI, a generalist AI designed to replace entire jobs with just prompts. Our agents typically run autonomously for hours, so effective context management is critical. I'd say that we invest most of our engineering effort into what is ultimately context management, such as:
1. Multi-agent orchestration
2. Summarising and chunking large tool and agent responses
3. Passing large context objects by reference between agents and tools
Two things to note that might be interesting to the community:
Firstly, when managing context, I recommend adding some evals to our context management flow, so you can measure effectiveness as you add improvements and changes.
For example, our evals will measure the impact of using Anthropics memory over time. Thus allowing our team to make a better informed decisions on that tools to use with our agents.
Secondly, there's a tradeoff not mentioned in this article: speed vs. accuracy. Faster summarisation (or 'compaction') comes at a cost of accuracy. If you want good compaction, it can be slow. Depending on the use case, you should adjust your compaction strategy accordingly. For example, (forgive my major generalisation), for consumer facing products speed is usually preferred over a bump in accuracy. However, in business accuracy is generally preferred over speed.
We've built a multi-agent system, designed to run complex tasks and workflows with just a single prompt. Prompts are written by non-technical people, can be 10+ pages long...
We've invested heavily in observability having quickly found that observability + evals are the cornerstone to a successful agent.
For example, a few things measure:
1. Task complexity (assessed by another LLM)
2. Success metrics given the task(s) (Agin by other LLMS)
3. Speed of agent runs & tools
4. Errors of tools, inc time outs.
5. How much summarizaiton and chunking occurs between agents and tool results
6. tokens used, cost
7. reasoning, model selected by our dynamic routing..
Thank god its been relatively cheap to build this in house.. our metrics dashboard is essentially a vibe coded react admin site.. but proves absolutely invaluable!
All of this happed after a heavy investment in agent orchestration, context management... it's been quite a ride!
Co-founder of Lopay here, we're a small but heavy Stripe user with £1B+ processed across Connect, Terminal, Identity, Instant payouts, Issuing... you name it.
We're looking at stable coins for the following use cases:
1. Instant clearing and settlement of 'floats' & liquidity - EG moving liquidity between our network to support instant/same day payouts or instant funding of a spend card.
2. Instant cross border payments (lots of people doing this already in companies that operate multinationally). EG, our USD top-ups today take 3 days in fiat, which can cause operational issues.
3. Offering our merchants (who are typically small businesses) optionality to hold USD in countries that have volatile currencies.
I'll also note that many people forget that the cost of a payment network isn't merely the movement of money, it's also KYC, dispute resolution, fraud prevention etc...
I wonder if the tempo team has looked at AI automating dispute resolution and fraud detection/prevention 'on chain'.. The network could fund the compute required for the AI to complete these tasks.
Cofounder of Lopay here - we have the same mission: offer free payments to businesses, but we're working with existing networks to do this.
QR code payments are particularly hard in countries like US and UK as you're trying to change consumer behaviour. I tried doing this in 2014 and again in 2019 - both failed to gain traction (aside from during COVID).
In the UK it's possible to accept card payments for 0% via Lopay, but only if you spend your earnings on our card (essentially, passing the fees onto the merchant/supplier you're paying). We're launching the same proposition in the US soon too.
If you don't use our card, our headline rate is 0.79%.
We're a lean team of just 36, supporting over 40k weekly transacting businesses with £1B+ in card processing. If anyone reading this is interested in this space, we're hiring and on the look out for driven people to join us!
Founder of Salamanca here, an app that aggregates every major restaurant booking platform into one app (OpenTable, SevenRooms, Tock, Resy, The Fork and others..)
Firstly, nice site - always love new tools to discover restaurants, thanks for posting, I’ve shared your blog post with friends, it was a brilliant read.
I have some recent experience working with restaurant reviews, I found that using only Google reviews can be unreliable, as some places that have top reviews may not be generally accepted as the ‘best’ restaurants.
We currently use a combination of Google reviews + Trip Advisor + Reviews from the booking platforms and we have web crawlers to check if the restaurant is featured on reputable restaurant guides or review sites.
We aggregate all of this review data and compute a “score”, so when users search for available tables in a city we can show available tables at the highest scoring restaurants first.
We apply Wilson score confidence intervals, to trust restaurant scores that have more reviews.
We are also applying an exponential decay when users list nearby restaurants, as you might be willing to travel a little further to go to a higher scoring restaurant.
Working with review data is fascinating.. we’re going to be launching an AI summary of recent reviews and our computed score in the coming weeks to help our users understand our ratings.
Our app went live on the App Store only a few days ago and we expect it to be live on Google play later this week.. so it’s an extremely busy time!
If you’re interested in what we’re doing please reach out, it would be great to connect, I really enjoyed your article!
It's called Zenning AI - we're a small team in London, testing it with a few companies at the moment!