First commit was 2 weeks ago. A clean-room rewrite of only the most essential APIs of litellm.
Was annoyed that DSPy was about 250 MB big. Mostly due to litellm being bloated and numpy. In order to create dspy-lite [1], I had to create a lighter litellm and make numpy optional (only essential for embeddings).
Made the lighter litellm litelm by selecting the parts of litellm's test suite that mattered and build it back to match litellms API.
Wrote a piece arguing that AI 1.0 is feudal because GPU scarcity concentrated power in a few cloud providers, but the coming GPU glut will invert this dynamic and make open weights + commodity compute the foundation of AI 2.0. The thesis is that sovereignty requires ownership of your stack (weights, compute, data) rather than renting intelligence from centralized APIs.
LLM-driven narrative game. Main technical issue is how go do compaction. I’ve devised a memory hierarchy that compacts the story to a constant amount of tokens per layer. Arc -> Scene -> Moment -> Line. Not sure if that’s the right dimensions to decompose into. Also tinkering how to get the right amount of “divergence” for story progression option generation. A lot of unanswered questions…
for me it feels like: Everything is stateful by default/convenience. Building robust systems is in part about confining statefulness to as few parts as possible. To contain statefulness. It’s to buy you some time and capacity. Yet the toughest problems often arise in the stateful parts of the system as well as quasi-stateless parts which sometimes develop hidden statefulness (think of syncing webclient and server state).
So being good at handling stateful systems is valuable. Maybe one should even embrace statefulness. However, the AWS Solution Architect will tell you otherwise.
This is a Great project I am really looking forward to using for internal apps. IMO a central requirement for internal apps is auth, especially OIDC/SAML-type auth. How do you recommend implementing this in mesop?
Was annoyed that DSPy was about 250 MB big. Mostly due to litellm being bloated and numpy. In order to create dspy-lite [1], I had to create a lighter litellm and make numpy optional (only essential for embeddings).
Made the lighter litellm litelm by selecting the parts of litellm's test suite that mattered and build it back to match litellms API.