The token reduction is a big deal. I've been building a tool that calls GitHub API via AI and token costs add up quickly. Curious how it handles pagination for large API responses?
This is impressive. I've been experimenting with Gemini API for a side project and the latency difference between local and cloud inference is something I keep thinking about. How does memory usage scale with the 500B models?