We are very happy that other developers also see the pain points of complex ETL and MLOps pipelines, specialized vector databases, and want to get rid of the need to move your data all over the place when building AI applications.
A big thank you to the community for the massive support!
However, achieving goals like "training your LLM" or enabling "real-time inference" requires more than just pipelines. For that, we have invested in enhancing compatibility with databases and facilitating parallel computing.
About your last point, I 'm not sure I fully understand. Do you mean to write a guide for moving lang-chain models to superduper? Or to create superduper wrappers for langchain ? Or to move the core functionalities of superduper to langchain ?
The guide, is something have in our immediate plans. The wrappers are under discussion. The latter I don't think it's possible due to architectural differences. For example, superduper is designed with multi-node environments in mind.
All things aside, it's a framework for building data workflows in Python.
Like taking the data from that source (e.g., SQL), processing them (e.g., pytorch or openai), and storing the results somewhere (e.g., data on Mongo metadata on SQL).
It actually consists of the following:
1. nifty abstractions for Data (e.g., sources, encoders, listeners), Metadata (e.g., vector indexes), Compute (e.g., sync, async, parallel).
2. gluing engine that transparently handles the interaction between components
3. out-of-the-box integrations with established tools (databases, AI models and APIs, compute engines)
This way, you can build customized data layers that sit on top of your database and save you from moving the data to dedicated systems (e.g., vector databases or MLops tools)