Show HN: LLMFlows – LangChain alternative for explicit and transparent apps(github.com)
github.com
Show HN: LLMFlows – LangChain alternative for explicit and transparent apps
https://github.com/stoyan-stoyanov/llmflows
22 comments
This looks nice!
I frequently am putting together complex chains of prompts and found microsoft guidance to be quite nice. Have you looked at it?
It feels like the abstraction I'd like the most is some way to combine few shot prompting, voting and chain-of-thought.
I have to create a prompt chain like this fairly often: write a description for this thing, given these previous input and outputs as examples (few shot). Do it N times with high randomness, maybe with expert personas, then review the solutions one by one with pros and cons (chain of thought) and then use all that to create a better description (final answer).
Right now, I always have to write out and connect all the steps, but its fairly rote and I think other prompting chains have a similar repetitiveness to them.
https://github.com/microsoft/guidance
I frequently am putting together complex chains of prompts and found microsoft guidance to be quite nice. Have you looked at it?
It feels like the abstraction I'd like the most is some way to combine few shot prompting, voting and chain-of-thought.
I have to create a prompt chain like this fairly often: write a description for this thing, given these previous input and outputs as examples (few shot). Do it N times with high randomness, maybe with expert personas, then review the solutions one by one with pros and cons (chain of thought) and then use all that to create a better description (final answer).
Right now, I always have to write out and connect all the steps, but its fairly rote and I think other prompting chains have a similar repetitiveness to them.
https://github.com/microsoft/guidance
Thank you for your feedback!
And thank you for sharing guidance!
You might find this guide useful to your use case:
https://llmflows.readthedocs.io/en/latest/user_guide/Async%2...
It shows how you can connect multiple similar flowsteps to a previous flowstep and they will all run in parallel.
You might find this guide useful to your use case:
https://llmflows.readthedocs.io/en/latest/user_guide/Async%2...
It shows how you can connect multiple similar flowsteps to a previous flowstep and they will all run in parallel.
Very cool!
https://feste.readthedocs.io/en/latest/design.html, this seems to be what Feste already does, and it offers automatic batching as well.
I'm really glad people are iterating on the abstraction for these systems. This is exactly the place where creativity is really useful
I see this only works with OpenAI at the moment. Any plans to include other sources, in thinking llama, bedrock.
Many of the open source inference solutions you'd use for llama ship with an openai-compatible interface (e.g. oobabooga/text-generation-web-ui, huggingface/text-generation-inference, etc). Probably can just change the API host and it would work.
Yes, I would love to get some feedback on what people want to see - right now I am thinking Bard and Claude. What do you want to have?
For gpt, both OpenAI and Azure OpenAI, I think they are similar implementation, you just have to provide some additional parameters like deployment name and different URL.
For non gpt options, I like Claude and Amazon titan. Claude is definitely good.
I'm also currently learning to use llama.cpp I see it has a langchain integration and it's very fast with GPU. So it's useful to be but as a hobbyist.
I've got no experience with bard though. Is it any good?
For non gpt options, I like Claude and Amazon titan. Claude is definitely good.
I'm also currently learning to use llama.cpp I see it has a langchain integration and it's very fast with GPU. So it's useful to be but as a hobbyist.
I've got no experience with bard though. Is it any good?
Thank you for your feedback. Azure OpenAI should be quite easy indeed. I’ve never tried Bard as well and have no clue how popular it is. Also I’ve never heard of Amazon Titan. Thank you for sharing!
Support for llama.cpp seems like it'd be really interesting for folks who don't want to send data back and forth to LLMaaS providers. https://github.com/ggerganov/llama.cpp
Makes sense! I will look into it
I think you've explained the "LangChain alternative" motivation pretty well, but as someone who hasn't yet worked with LLMs from a development perspective, what are the primary benefits of a framework like this?
Does the framework help with building the vector store? If not, are there complementary frameworks you use and/or would recommend? I'd think one would want something that watches a collection of source documents (with lots of supported formats) and updates the vector store as those change.
One of the primary benefits of a framework seems to be to act as an abstraction for LLMs and vector stores. I see LLMFlows supports Pinecone. Does it support others (e.g. pgvector)? Does it support LLMs other than OpenAI LLMs?
Does the framework help with building the vector store? If not, are there complementary frameworks you use and/or would recommend? I'd think one would want something that watches a collection of source documents (with lots of supported formats) and updates the vector store as those change.
One of the primary benefits of a framework seems to be to act as an abstraction for LLMs and vector stores. I see LLMFlows supports Pinecone. Does it support others (e.g. pgvector)? Does it support LLMs other than OpenAI LLMs?
Thank you for your feedback!
Building apps with LLMs is fundamentally an exercise of manipulating strings and making API calls (Both LLM and vector db APIs). When apps are more complex and start including multiple LLM calls and prompts that can change dynamically some additional challenges start to emerge. e.g figuring out:
- When was a particular LLM called?
- How much time did it take?
- What were the input variables?
- What did the prompt look like?
- What was the exact configuration of each LLM?
- How many times did we retry the request?
- What was the raw data the API returned?
- What was exactly returned from the vector store and fed into an LLM?
- How many tokens were used?
- What was the final result for each call?
- How do you make API calls in parallel?
I think a framework like this should provide abstractions that allow people to focus on the important part like prompt engineering and productionizing the app and worry less about how to figure out stuff like this.
Right now LLMFlows supports only OpenAI models and Pinecone but I am working on classes for Chroma and Weaviate. And I would like to also provide support for Bard and Claude once I get access.
Building apps with LLMs is fundamentally an exercise of manipulating strings and making API calls (Both LLM and vector db APIs). When apps are more complex and start including multiple LLM calls and prompts that can change dynamically some additional challenges start to emerge. e.g figuring out:
- When was a particular LLM called?
- How much time did it take?
- What were the input variables?
- What did the prompt look like?
- What was the exact configuration of each LLM?
- How many times did we retry the request?
- What was the raw data the API returned?
- What was exactly returned from the vector store and fed into an LLM?
- How many tokens were used?
- What was the final result for each call?
- How do you make API calls in parallel?
I think a framework like this should provide abstractions that allow people to focus on the important part like prompt engineering and productionizing the app and worry less about how to figure out stuff like this.
Right now LLMFlows supports only OpenAI models and Pinecone but I am working on classes for Chroma and Weaviate. And I would like to also provide support for Bard and Claude once I get access.
> I think a framework like this should provide abstractions that allow people to focus on the important part like prompt engineering and productionizing the app and worry less about how to figure out stuff like this.
I appreciate the detailed explanation, thank you!
If you'll permit another noob question, can/should frameworks like this include source attribution for responses? Based on articles like https://jamesg.blog/2023/04/02/llm-prompts-source-attributio... I'd guess the strict answer would be no, and that any strategies you might use to elicit them might hallucinate citations.
I appreciate the detailed explanation, thank you!
If you'll permit another noob question, can/should frameworks like this include source attribution for responses? Based on articles like https://jamesg.blog/2023/04/02/llm-prompts-source-attributio... I'd guess the strict answer would be no, and that any strategies you might use to elicit them might hallucinate citations.
This is a very hard problem. If you base your response on a text that has been returned from a vector DB you can reference this text. But if you rely on a citation generated directly from the LLM is probably impossible to reliably guarantee the source is real. I guess if it’s a link you can check if it’s actually a real link, scrape the content and do some extra trickery to figure out if it contains the text produced by the LLM but I’ve never tried this myself.
> If you base your response on a text that has been returned from a vector DB you can reference this text.
Oh, sweet! That would be a wonderful feature to have in a framework. For personal or corporate knowledge bases, sources would be really helpful for people who want to dig deeper.
Oh, sweet! That would be a wonderful feature to have in a framework. For personal or corporate knowledge bases, sources would be really helpful for people who want to dig deeper.
> What did the prompt look like?
This is the main thing I'm struggling with langchain. I needed to see this info for troubleshooting. I don't know if it's easily visible and I'm being thick, or it's just hidden away.
This is the main thing I'm struggling with langchain. I needed to see this info for troubleshooting. I don't know if it's easily visible and I'm being thick, or it's just hidden away.
There's been a lot of discussion on the shortcomings of langchain in the past few weeks, but when I first tried it in March, I thought there are 3 main problems: 1. Too many abstractions 2. Hidden prompts and opinionated logic in chains which makes it hard to customize 3. Hard to debug
This inspired me to try and build a framework that solves these 3 issues, and therefore I started building LLFlows with the "philosophy" of being "simple, explicit, and transparent."
A few weekends later, I think I finally managed to reach a state where I feel it's ready to be shared.
I would love to hear your feedback! Thank you!