It's local first, and ties many different AIs into one text editor, any arbitrary text editor in fact.
It does speech recognition, which isn't useful for writing code, but is useful for generating natural language LLM prompts and comments.
It does CodeLlama (and any HuggingFace-based language model)
It does ChatGPT
It does Retrieval Augmented Gen, which is where you have a query that searches through eg PDFs, Youtube transcripts, code bases, HTML, local or online files, Arxiv papers, etc. It then surfaces passages relevant to your query, that you can then further use in conjunction with an LLM.
I don't know how mainstream LLM-powered software looks, but for devs, I love this format of tying in the best models as they're released into one central repo where they can all play off each others' strengths.
I've done the same but was very disappointed with the stock sentence embedding results. You can get any arbitrary embedding, but then the cosine similarity used for nearest neighbor lookup gives a lot of false pos/negs.
*There are 2 reasons:*
1. All embeddings from these models occupy a narrow cone of the total embedding space. Check out the cos sim of any 2 arbitrary strings. It'll be incredibly high! Even for gibberish and sensical sentences.
2. The dataset these SentenceTransformers are trained on don't include much code, and certainly not intentionally. At least I haven't found a code focused one yet.
*There are solutions I've tried with mixed results:*
1. embedding "whitening" forces all the embeddings to be nearly orthogonal, meaning decorrelated. If you truncate the whitened embeddings, and keep just the top n eigenvalues, you get a sort of semantic compression that improves results.
2. train a super light neural net on your codebase's embeddings (takes seconds to train with a few layers) to improve nearest neighbor results. I suspect this helps because it rebiases learning to distinguish just among your codebase's embeddings.
*There are solutions from the literature I am working on next that I find conceptually more promising:*
1. Chunk the codebase, and ask an LLM on each chunk to "generate a question to which this code is the answer". Then do natural language lookup on the question, and return the code for it.
2. You have your code lookup query. Ask an LLM to "generate a fabricated answer to this question". Then embed it's answer, and use that to do your lookup.
3. We use the AST of the code to further inform embeddings.
I have this in my project UniteAI [1] and would love if you cared to collab on improving it (either directly, or via your repo and then building a dependency to it into UniteAI). I'm actually trying to collab more, so, this offer goes to anyone! I think for the future of AI to be owned by us, we do that through these local-first projects and building strong communities.
If you're ok working in a text editor, UniteAI works on pdfs, youtube transcripts, code repos, web pages, local documents, etc. The nice thing about the editor is once it's done retrieval, you can hit another keycombo to send retrieved passages to an LLM (local, or chatgpt), and ask questions or favors about it (such as summarization, or formatting changes).
TL;DR: chat with AI, code with AI, speak to AI (voice-to-text + vice versa), have AI search huge corpora or websites for you, all via an interface of collaborating on a text doc together in the editor you use now.
Motivation
I find the last year of AI incredibly heartening. Researchers are still regularly releasing SoTA models in disparate domains. Meta is releasing powerful Llama under generous provisions (As is the UAE with Falcon?!). And the Open Source community has shown a tidal wave of interest and effort into building things out of these tools (112k repos on GH mentioning ML!).
Facing this deluge of valuable things that communities are shepherding into the world, I wanted to incorporate them into my workflows, which as a software engineer, means my text editor.
* Ties in to any editor via Language Server Protocol. Like collaborating in G-Docs, you collab with whatever AI directly in the document, all of you writing alongside each other concurrently.
* Like Copilot / Cursor, it can write code/text right in your doc.
* It supports any Locally runnable model (Llama family, Falcon, Finetunes, the 21k available models on HF, etc.)
* It supports OpenAI/ChatGPT via API key.
* Speech-to-Text, useful for writing prompts to your LLM
* You can do Semantic Search (Retrieval Augmented Generation) on many sources: local files, Arxiv, youtube transcripts, Project Gutenberg books, any online HTML, basically if you give it a URI, it can probably use it.
* Written in Python, so, much more generic than writing a bespoke `some_specific_editor` plugin.
Caveat
Since it always comes up, AI is not perfect. AI is a tool to augment your time, not replace it. It hallucinates, it lies, it bullshits, it writes bad code, it gives dangerous advice.
But can still do many useful things, and for me it is a huge force multiplier.
You need a Human In The Loop, which is why it's nice to work together iteratively on a text document, per, this project. You keep it on track.
Why is this interesting
These tools play well when used together:
* Code example: I can Voice-to-Text a function comment then send that to an LLM to write the function.
* Code example 2: I can chit chat about project architecture plans, and strategies, and libraries I should consider.
* Documentation example: I can retrieve relevant sections of my city's building code with a natural language query, then send that to an LLM to expound upon.
* Authorship example: I can have my story arcs and character dossiers in some markdown file, and use that guidance to contextualize an AI as it works with me for writing a story.
* Entertainment example: I told my AI it was a Dungeon Master, then over breakfast with friends, used Voice-to-Text and Text-to-Wizened-Wizard-Voice, and played a hillarious game. I still had to drive all this via a text doc, and handy key combos.
This is still nascent, and I welcome all feedback, positive or critical.
We have a community linked on the repo which you're invited to join.
I'd love love to chat with people who like this idea, use it, want to see other features, want to contribute their effort, want to file bug reports, etc.
A big part of my motivation in this is to socialize with like-minds, and build something cool.
This is my project, where the goal is to Unite your AI-stack inside your editor (so, Speech-to-text, Local LLMs, Chat GPT, Retrieval Augmented Gen, etc).
It's built atop a Language Server, so, while no one has made an IntelliJ client yet, it's simple to. I'll help you do it if you make a GH Issue!
It's local first, and ties many different AIs into one text editor, any arbitrary text editor in fact.
It does speech recognition, which isn't useful for writing code, but is useful for generating natural language LLM prompts and comments.
It does CodeLlama (and any HuggingFace-based language model)
It does ChatGPT
It does Retrieval Augmented Gen, which is where you have a query that searches through eg PDFs, Youtube transcripts, code bases, HTML, local or online files, Arxiv papers, etc. It then surfaces passages relevant to your query, that you can then further use in conjunction with an LLM.
I don't know how mainstream LLM-powered software looks, but for devs, I love this format of tying in the best models as they're released into one central repo where they can all play off each others' strengths.