Ask HN: What does your AI tech stack look like?
14 comments
I am all local
VoltaML for stable diffusion imagegen (as AITemplate is really fast, and controlnet and LORAs are super useful)
Llama.cpp for textgen... I use the koboldcpp frontend (and sometimed AI Horde) for experimenting with models, though I am still figuring out the exact stack I want for productivity. I am watching this PR closely:
https://github.com/ggerganov/llama.cpp/pull/1773
As well as Starcoder/Openllama finetunes for summarization, codegen and such.
VoltaML for stable diffusion imagegen (as AITemplate is really fast, and controlnet and LORAs are super useful)
Llama.cpp for textgen... I use the koboldcpp frontend (and sometimed AI Horde) for experimenting with models, though I am still figuring out the exact stack I want for productivity. I am watching this PR closely:
https://github.com/ggerganov/llama.cpp/pull/1773
As well as Starcoder/Openllama finetunes for summarization, codegen and such.
Oh, nice flag on that PR Thanks for surfacing that!
I've been hearing a lot about AITemplate on Twitter. Seems the consensus is it's hella quick.
I've been hearing a lot about AITemplate on Twitter. Seems the consensus is it's hella quick.
Yeah, its insanely fast compared to PyTorch triton. And it supports dynamic input too.
I think its overlooked in Stable Diffusion land because its (last I checked) linux/wsl only and HF diffusers based.
I think its overlooked in Stable Diffusion land because its (last I checked) linux/wsl only and HF diffusers based.
for text, is there a standard to compare model results?
There are tons of metrics people have come up with, for example look at the huggingface leaderboard. There are more niche leaderboards/tests for chat models, chain of thought, summarization and such.
But the best test is personal experimentation. Prompt engineering and subjective preference have a massive effect on finetune performance.
But the best test is personal experimentation. Prompt engineering and subjective preference have a massive effect on finetune performance.
OpenAI + PineconeDB -> Langchain -> Quivr to get a decent UI
right now i'm trying to prove to the company i work that you can turn the endless rules and daily information stream into something people can find easily using sematic search (vector DB) and an AI to summarize categorize, generate embeddings and auto update the DB, while also interpreting the content of the search result and giving a more digestible answer and a source link if the end user needs more information.
Keyword based search is a pain when different products have similar names or the internal search doesn't filter words like "of" and "and" from the keyword search.
After that, if the tests work and i can push the idea forward, it'll probably be OpenAI + a local vector DB (Chroma?) + custom made search page for internal usage
right now i'm trying to prove to the company i work that you can turn the endless rules and daily information stream into something people can find easily using sematic search (vector DB) and an AI to summarize categorize, generate embeddings and auto update the DB, while also interpreting the content of the search result and giving a more digestible answer and a source link if the end user needs more information.
Keyword based search is a pain when different products have similar names or the internal search doesn't filter words like "of" and "and" from the keyword search.
After that, if the tests work and i can push the idea forward, it'll probably be OpenAI + a local vector DB (Chroma?) + custom made search page for internal usage
Or you can try a proper open-source vector db like Qdrant :)https://github.com/qdrant/qdrant
Thoughts on fine tuning pythia, flan-t5 or codegen/salesforce models? Possibly to run the model and DB locally?
would love to talk more about this as im working on something similar... anywhere i can reach you?
For us the complexity has all been about the infra around the AI stack. For example, having a k8s cluster for running inference for open source models (Stable Diffusion, Bark, etc.). Beyond that it’s been custom abstractions around Open AI/PaLM APIs and vector stores (pinecone and Faiss).
This stack is evolving so quickly but we are convinced it goes back to a distributed systems/infra problems when it comes to complexity, unless you’re training your own foundation models.
This stack is evolving so quickly but we are convinced it goes back to a distributed systems/infra problems when it comes to complexity, unless you’re training your own foundation models.
A deeper look into our stack if you're interested: https://www.commandbar.com/blog/harnessing-ai-magic
Right now I am playing with modal.com, trying to run some pytorch code lol. Not sure if that counts as I am just learning.
Nice one What are you building?
Just trying to get nanoGPT running on it first. I got the colab version working now trying the github version that has more features and knobs to twiddle
Some of our favorites and more prolifically used were OpenAI API, ChatGPT, Midjourney, and Jasper.
Curious to hear what other AI tools (and frameworks like LangChain etc) y'all are heavily dependent on