for pass@1 HumanEval tells how well the model solves a task from a set, given only one chance to solve it. It's not the perfect metric, there're other like DS-1000, MBPP (we have included them on HuggingFace model card). HumanEval is good for benchmarking with other models as it gives a fast idea how powerful the model is.
we want to help developers who need either on-premise or permissive code assistant, copilot has neither of this. We also wanted to lower the barriers for self-hosting, so that the model is available on most GPUs with just 3GB Ram. Plus making the code completions fast and efficient (understanding entire context, not just the previous tokens).
We’ve finished training a new code model Refact LLM which took us about a month. The main use-case is for blazing-fast code completion with fill-in-the-middle, additionally, the model could reply to chat prompts.
It has much better performance than all of the code models of similar size, and almost reaches the same HumanEval as Starcoder being 10x smaller in size.
With the small size, it can work with most modern GPUs requiring just 3GB Ram.
we try to eliminate this problem by using code models trained only on permissevely licensed code, then you can run them locally without sending code anywhere
we're going in this direction for code models with Refact https://github.com/smallcloudai/refact/ - right now you self-host code models, fine-tune them on local files, get the model running locally inside your IDE
We're building the best open-source AI for coding using all the best LLMs available in the internet (GPT, Starcoder, WizardCoder, LLama2).
We’re hiring for Refact’s first Developer Advocate. As the first Developer Advocate, you'll play a key role in helping developers use the power of AI for coding. You'll have the opportunity to grow and support a community of developers who are excited about the future of AI and open-source.
We're looking for a Software Engineer to help us build the best developer experience for the AI code assistants inside IDEs. We currently have VS Code and Jetbrains plugins and plan to add more IDEs and more functionality to the plugins.
Hi HN! At Refact, we're building an open-source AI Code assistant with fine-tuning focused on providing the enjoyable coding experience without privacy concerns.
Why? When you look at AI code tools like Copilot, you'll notice that they often provide generic coding suggestions because the models were not trained on your codebase. They also only work in the cloud, so in order to get a code suggestion, you need to send your code to the cloud provider.
One of the solutions for this is self-hosted fine-tuning on codebase, but the companies who provide this feature currently prioritize it for enterprise making it difficult for individual devs to use it.
We want to change that by open-sourcing fine-tuning for everyone, not just enterprise customers.
Currently with Refact you can self-host Refact models and the best open-source LLMs (like StarCoder, LLama2, WizardLM) and use it for code suggestions and chat.
We would love to hear your ideas and feedback on your current experience with AI code assistants and what is currently missing!
I'm excited to see all the model's implementations that are yet to come. Refact.ai seems to be working on code transformation tools and chat inside Jetbrains and VSCode powered by StarCoder https://refact.ai/blog/2023/self-hosted-15b-code-model/
while you're waiting, maybe you'll be interested in checking https://refact.ai it's an open-source alternative to Copilot X with features like code transformation and integrated chat on top of code completion