Show HN: Llmbuffer – Python library for cache-optimized LLM conversation history(github.com)

7 pointsby scottmp10पिछला माह1 comments

Show HN: Llmbuffer – Python library for cache-optimized LLM conversation history

https://github.com/scottpurdy/llmbuffer

I was not getting good cache utilization when including dynamic context in agent threads. After a lot of experimentation, I found a good pattern that minimizes how often long lived conversation history gets modified while still supporting dynamic context. It has flexible hooks for doing things like truncating or summarizing tool outputs when transitioning messages to the long term history. And I'm seeing >>90% of tokens hitting the cache for my agents despite including a lot of dynamic user context.

There are a wide range of agent prompting strategies so I'd love to hear where this library works well and where there are patterns that don't fit well into the current API!

2 comments

tam15929 दिन पहले

Cool! But I think the order misses the 1st one: tool set definition

sapsapshenपिछला माह