> tool call pruning breaks cache and people will tell you this is horrible and expensive
> except i looked at some anthropic data and real user behavior ends up with better cache hits and 30% less spend
> even this is needs to be analyzed further, it's just not simple
> for openai data it's inverted! cache hit ratio is actually better [sic: I think he meant worse based on the screenshot] with tool call pruning turned on
> but the net $ saved is only 5%
> kimi is a funny one - it has better cache hits with pruning on...but is also more expensive!
There was also another thread recently where he discussed that pruning improves user experience (models are smarter with less context) but I can't find it.
Again, did you read the order? The judge's order explicitly said this would be legal and cites the law permitting it, then goes on to explain why this action did not satisfy it:
> Covered procurement actions include “[t]he decision to withhold consent for a contractor to subcontract with a particular source or to direct a contractor . . . to exclude a particular source from consideration for a subcontract.” 10 U.S.C. § 3252(d)(2)(C).
They can say "sorry Palantir, we will only sign a contract with you if you commit not to use Claude to provide services" and then Palantir is free to decide if they want to accept the terms of the contract or not. This is how business works.
Did you read the order? It directly addresses your comment:
> More importantly, as discussed above, no one is entitled to conduct business with the Federal Government, see Perkins, 310 U.S. at 127, and irrespective of the challenged actions, DoW and other federal agencies are free to terminate its contracts and agreements with Anthropic, as Anthropic readily admits.
It essentially depends on how many back-and-forth calls are required. If the model returns a request for multiple calls at once, then the reply can contain all responses and you only pay once.
If the model requests tool calls one-by-one (e.g. because it needs to see the response from the previous call before deciding on the next) then you have to pay for each back-and-forth.
If you look at popular coding harnesses, they all use careful prompting to try to encourage models to do the former as much as possible. For example opencode shouts "USING THE BATCH TOOL WILL MAKE THE USER HAPPY" [1] and even tells the model it did a good job when it uses it [2].
That's correct. Input caching helps, but even then at e.g. 800k tokens with all of them cached, the API price is $0.50 * 0.8 = $0.40 per request, which adds up really fast. A "request" can be e.g. a single tool call response, so you can easily end up making many $0.40 requests per minute.
An interesting thing to understand about Klarna and other buy-now-pay-later products is that a major part of their profit is the very high merchant fees they charge; retailers have to pay ~2-4x what they do for credit cards if they want to offer Klarna. 57% of Klarna's profit comes from these merchant fees compared to just 24% from loan interest [1].
It turns out it's worth it to merchants because when you're not paying now, you end up buying more than you would otherwise. Order sizes are ~15% higher [2]. Probably similar to how it hurts more to pay with cash than debit because it's so tangible.
I view it kinda similar to gambling apps with their endlessly optimized special offers designed to exploit the human monkey brain.
> tool call pruning breaks cache and people will tell you this is horrible and expensive
> except i looked at some anthropic data and real user behavior ends up with better cache hits and 30% less spend
> even this is needs to be analyzed further, it's just not simple
> for openai data it's inverted! cache hit ratio is actually better [sic: I think he meant worse based on the screenshot] with tool call pruning turned on
> but the net $ saved is only 5%
> kimi is a funny one - it has better cache hits with pruning on...but is also more expensive!
There was also another thread recently where he discussed that pruning improves user experience (models are smarter with less context) but I can't find it.
This can also be disabled in the config: https://opencode.ai/docs/config/#compaction