Next-generation AI models(tabnine.com)
tabnine.com
Next-generation AI models
https://www.tabnine.com/blog/announcing-tabnine-next-generation/
138 comments
Call me old fashioned, but I use my brain for mutli-line code generation
Great, you are the best, super cool! Go eat an apple
I'm just describing what all programmers have done since the beginning of time
Real programmers use punch cards for programming.
You don't see how you just shot yourself in the foot, do you?
If nothing else the HN mantra is not "That's how we've always done it."
If nothing else the HN mantra is not "That's how we've always done it."
The entire programmer ethos revolves around building deep understandings of the level abstraction you're working at and deep diving into the layers below when necessary or even just for fun!
I've watched people use co-pilot but and I'm not 100% convinced it's a good thing but I'm open to it. The problem I've seen is that in some cases it generates some really bad code, code that could've been replaced by usage of a standard library.
What I think it's really good at is helping to learn new languages. Getting idiomatic code generated, even if you throw it out eventually is a great way to get a feel for a new language without having to jump back and forth between the editor and the browser.
I've watched people use co-pilot but and I'm not 100% convinced it's a good thing but I'm open to it. The problem I've seen is that in some cases it generates some really bad code, code that could've been replaced by usage of a standard library.
What I think it's really good at is helping to learn new languages. Getting idiomatic code generated, even if you throw it out eventually is a great way to get a feel for a new language without having to jump back and forth between the editor and the browser.
I save tons of time having to write matplotlib code etc. because AI does it for me and I can spend more time thinking about the deep parts of research. I basically haven't had to write more than 1 or 2 keystrokes for a plot I've made in months now. If that's not useful I don't know what is.
that sounds amazing, I just spent 3 hours last night making mpl plots for ideas I already had. this sounds like a perfect use case for me
I see it differently. The entire ethos should revolve around product that works. Product the meets a market need. Product that brings joy (not friction) into the world.
Any tool that can meet that ends should be considered.
Put another way, users don't care what anyone (i.e., in engineering) does or does not know. They're not going to wake up and say "I only want to use products that don't use Copilot." Nobody cares about any ethos, etc.
Any tool that can meet that ends should be considered.
Put another way, users don't care what anyone (i.e., in engineering) does or does not know. They're not going to wake up and say "I only want to use products that don't use Copilot." Nobody cares about any ethos, etc.
Well I don't think I've shot myself in the foot, it seems a lot of people agree with me.
I'm not against the idea of ai coding tools on premise, but it did strike me as kind of funny to imagine someone using two such tools at the same time.
I'm more of a minimalist, and would generally prefer to have lean tools which get out of my way. The idea that every keystroke involves a back-and fourth with a server, and that my basic coding workflow would need an internet connection is not something which is super appealing to me personally.
I'm not against the idea of ai coding tools on premise, but it did strike me as kind of funny to imagine someone using two such tools at the same time.
I'm more of a minimalist, and would generally prefer to have lean tools which get out of my way. The idea that every keystroke involves a back-and fourth with a server, and that my basic coding workflow would need an internet connection is not something which is super appealing to me personally.
Why? They clearly explained how it works for them. What's funny is you not being open to new ideas. That's a worse sign for us all than Copilot.
[deleted]
[deleted]
Yikes
Doctors beware.
This is the attitude. Yeah.
Call me old fashioned, but when I need to create so much boilerplate that code generation starts to make sense, I usually just try to solve the problem in another more concise way.
Yeah exactly - in software development the problem is rarely that there is too much typing to do.
I used to use that, but Copilot works better with multiple languages!
...just kinda joking: for me, one major use case of this thing is a faster alternative to googling some syntax or method name.
...just kinda joking: for me, one major use case of this thing is a faster alternative to googling some syntax or method name.
As with any technology, there are those who don't like it, and that's fine.
I personally don't "trust" AI's decisions all that much, and I enjoy learning how to implement things on my own. Maybe that's why I won't be using Copilot, at least for now.
I feel Copilot indispensable - with the constant emergence of frameworks, languages I need to keep learning, it really shortens the time it takes to get productive in an unfamiliar environment.
It's existed for less than a year and you find it indispensable?
I don't see the contradiction - if someone who started programming in the pre-Internet days in a text editor, then switched to an IDE with autocomplete and searchable docs, would it be hyperbole for him to say that he finds these tools indispensable?
You are not the only one that has said this but would love a link to a video on how you like to work with both together.
Copilot uses a huge model and a ton of GPU's (and an unlimited budget) to focus on really long completions. This is a very specific choice to go for long completions.
We have worked hard to solve this problem differently and with the new generation you will see longer multi-line completions while also giving developers the options to use shorter suggestions (perhaps together with Copilot)
Copilot uses a huge model and a ton of GPU's (and an unlimited budget) to focus on really long completions. This is a very specific choice to go for long completions.
We have worked hard to solve this problem differently and with the new generation you will see longer multi-line completions while also giving developers the options to use shorter suggestions (perhaps together with Copilot)
Having just purchased copilot and a user of Tabnine in the past (using Phpstorm) here is my own personal opinion. The suggestions offered by Tabnine weren't anywhere good as copilot - not even by a long shot (at least for PHP and Javascript).
But almost an year has passed since I used this so I don't know if they have had a major update and the results are improved, but previously 1 out of 10 tabnine suggestions were barely useful, compared to 9 out of 10 with copilot.
Since it's a paid offering now I really hope they've upped their game.
But almost an year has passed since I used this so I don't know if they have had a major update and the results are improved, but previously 1 out of 10 tabnine suggestions were barely useful, compared to 9 out of 10 with copilot.
Since it's a paid offering now I really hope they've upped their game.
Hi, Tabnine's DevRel here.
The past 12 months have been quite a journey, and a lot has changed. To make a long story short: We've trained 12 standalone models that are now used to serve highly accurate code completions based on current project & file context. I encourage you to give it another go.
Tabnine CEO Dror Weiss, wrote this detailed thread explaining some of the key differences (both in technology and user-approach) between Tabnine and Copilot: https://twitter.com/drorwe/status/1539293063117516801
Tabnine CEO Dror Weiss, wrote this detailed thread explaining some of the key differences (both in technology and user-approach) between Tabnine and Copilot: https://twitter.com/drorwe/status/1539293063117516801
fartcannon(5)
I remember when Jacob Jackson’s (this kid is nothing less of a genius) tabnine project was posted on HN. Codota then bought tabnine and now they renamed themselves after this acquisition. Got me interested what Jacob is doing now. Looks like after his stint at OpenAI this is his new project: https://same.energy/
Thanks for linking it. I love the concept! I was looking for some art for the wall and I can't think of a better way to explore the options when you don't have a specific idea about the result you're after.
Jacob Jackson is awesome and we loved the time we got to work with him. Brilliant dude
My experience with Tabnine was that I loved it when it came out. I loved the fact that I could get full line suggestions without any extra language specific setup (I hop around different languages a lot) but the suggestions became less and less useful over time. In my view this was a degradation of Tabnine but of course I have nothing to back this up. Hearing this backstory does reaffirm my suspicion.
At some point I started using Language Server (mostly Typescript and Python) suggestions alongside Tabnine and it started to feel like it was just getting in the way. In the end I got rid of Tabnine. I haven't tried Co-Pilot yet but may give it a go when I find some time.
At some point I started using Language Server (mostly Typescript and Python) suggestions alongside Tabnine and it started to feel like it was just getting in the way. In the end I got rid of Tabnine. I haven't tried Co-Pilot yet but may give it a go when I find some time.
I really want to love Tabine. I really, really do.
I gave it a fair chance, let it index my private personal projects, ran it on my personal machine to see how it worked. The results are... not great. I'm using IntelliJ, and Tabnine is arrogant enough to believe that their autocomplete is superior to the libraries autocomplete, even when completely wrong or it has no idea of its autocomplete. These are opensource libraries.
Their inline suggestions (per what Github Copilot) isn't great either. I've tried their different strength levels (their UI for their product is fantastic) and it's not great.
Github copilot when I first started was pretty good, over the last few months it's become amazing. I'm going to be giving my money to Github over Tabnine, but would prefer to give it to Tabnine.
I gave it a fair chance, let it index my private personal projects, ran it on my personal machine to see how it worked. The results are... not great. I'm using IntelliJ, and Tabnine is arrogant enough to believe that their autocomplete is superior to the libraries autocomplete, even when completely wrong or it has no idea of its autocomplete. These are opensource libraries.
Their inline suggestions (per what Github Copilot) isn't great either. I've tried their different strength levels (their UI for their product is fantastic) and it's not great.
Github copilot when I first started was pretty good, over the last few months it's become amazing. I'm going to be giving my money to Github over Tabnine, but would prefer to give it to Tabnine.
Seconded. Tabnine's suggestions were way worse than IntelliJ's suggestions. It was very annoying. Uninstalled it after a giving it a good try.
Java was the one language where I found TabNine to fall short. I think it’s a harder case for AI completions, because the existing story for editing Java is a lot less like editing text than other languages.
I haven’t tried Copilot for editing Java, because I’m no longer working in any Java code bases, but I wonder if it does a better job.
I haven’t tried Copilot for editing Java, because I’m no longer working in any Java code bases, but I wonder if it does a better job.
I uninstalled T9 for this reason: bad suggestions superseding the regular autocomplete, this is just not good.
I have yet to compare to copilot.
I have yet to compare to copilot.
Same issue with the autocompletes pushing the ground-truth out.
I actually installed a third-party extension which implements their autocomplete differently.
So weird that they didn't hear such major/consistent feedback. (Maybe they didn't want to tank their 'suggestions taken' number?)
I actually installed a third-party extension which implements their autocomplete differently.
So weird that they didn't hear such major/consistent feedback. (Maybe they didn't want to tank their 'suggestions taken' number?)
please take a look at inline suggestions that will be in grey and not in the regular autocomplete box. It is now the default setup.
Same here. I gave it a go for a while, thinking I must be holding it wrong or something.
Whenever I encounter tabnine pages in search results- most commonly for Java Spring stuff- and click through, I regret it. The code snippets are never useful and the ads/popups are offensive. Baeldung resources are infinitely better.
Yeah, I was trying to find a polite way of pointing out that even their screen capture showing how "awesome" the completion is is almost guaranteed to be wrong
File fileName =
File fileName = new File(path);
no need to worry we love good feedback here at Tabnine ;-) Your observations are appreciated. In this case we are using general code and not a custom model but if you would like a custom model then I would wager you would get better variable names... maybe even "awesome".
Our goal with our customers is to build models based on the best custom code and input from the best developers so that your new devs can learn from YOUR CODE and with YOUR BEST PRACTICES.
When looking across a single companies codebase, how will you distinguish "best practices" from "we know this is bad but it works don't break it"? I have yet to work somewhere where the former is more than 5-10% of the code in production so I worry about this making it harder to raise the bar.
I would find it a more compelling pitch to say tabnine will help me and my teammates follow more general best practices, though I understand how subjective that is.
I would find it a more compelling pitch to say tabnine will help me and my teammates follow more general best practices, though I understand how subjective that is.
A model is only as god as the code that it trains on and the people it learns from. In your example above we can build a custom model on "best practices" code and not the companies codebase. Our work here has focused on working with ISV's like AWS, GCP, etc as they will have best practices for their tech. We then pull that together for a team in a custom model.
Cool, that makes sense.
Do you models incorporate any sort of time-based weighting, where more recent code is weighted more heavily? Seems like that would also help this issue, as more recent code tends to be closer to what we hope future code looks like. But then how do you let best practices evolve over time?
Do you models incorporate any sort of time-based weighting, where more recent code is weighted more heavily? Seems like that would also help this issue, as more recent code tends to be closer to what we hope future code looks like. But then how do you let best practices evolve over time?
I had to uninstall Tabnine because its resource usage was unacceptable. Hopefully they've addressed that, because it was using way too much of my CPU at times that it shouldn't have been.
I used to have limited memory (8gb for my wsl container) and shit went crazy (had a f_u_tabnine bash function that ran a loop to kill the tab nine procs that had cores pinned) to the point of me giving up ~2 years ago. Now on 16gb memory for a wsl container, I've had no such problems with tabnine. So not clear if there were fixes or if more memory helped.
Hi, Tabnine's DevRel here.
I'm glad to say that we have addressed this issue: You can now easily opt-in to encrypted cloud mode through the Tabnine hub (find it under Public Model section). This option takes off most of the computing resources used to run Tabnine. That's just one out of many other improvements we've been working on lately, that are now generally available. I hope you'll give it another go! :)
Even after I enabled cloud model just today, the local binaries are still taking a lot of CPU resources in bursts of say once every four seconds, I have the latest versions of both the tabnine extension and vscode.
feel free to turn off local completely. The bursts you are seeing is building a local transformer model on your computer. For enterprises and developers that dont want their data to go anywhere this is critical but does come with tradeoffs. We continue to make this better or feel free to use with GitPod (https://www.gitpod.io/blog/tabnine)
For my curiosity, https://www.tabnine.com/code/java/classes/com.google.gson.Js... says it was taken from https://github.com/Vedenin/useful-java-links/tree/master/hel... which Tabnine annotates as "license: other" (it's actually CC-BY-SA-4.0 <https://github.com/Vedenin/useful-java-links/blob/1f4278c9ad...>) but (a) does Tabnine not know about CC licenses? (b) how is the downstream user (i.e. the person accepting said snippet completion) supposed to interpret "license: other" without doing the same research I just did?
That story gets even weirder with the 3rd link on that page whose license is also "other" but this time the "view source" link goes to https://www.tabnine.com/web/assistant/code/rs/5c781237e70f87... . I find that weird for at least two reasons: (a) it clearly says "This snippet was taken from github" and has a GitHub style "org/repo" nomenclature, but doesn't link to the actual repo (b) at the very top of that file is the boilerplate Apache 2.0 license header
Finally, one should be very cautious about ever linking to "master" URLs, since the branch can get nuked if the repo owner decides to go with the "master to main" rename, it can lead the user to a copy of the file that is almost guaranteed not to be the same sha as the one Tabnine indexed, and related to that the repo can undergo a license change (FOSS to BSL is a very common one) leading to some complicated discussions
That story gets even weirder with the 3rd link on that page whose license is also "other" but this time the "view source" link goes to https://www.tabnine.com/web/assistant/code/rs/5c781237e70f87... . I find that weird for at least two reasons: (a) it clearly says "This snippet was taken from github" and has a GitHub style "org/repo" nomenclature, but doesn't link to the actual repo (b) at the very top of that file is the boilerplate Apache 2.0 license header
Finally, one should be very cautious about ever linking to "master" URLs, since the branch can get nuked if the repo owner decides to go with the "master to main" rename, it can lead the user to a copy of the file that is almost guaranteed not to be the same sha as the one Tabnine indexed, and related to that the repo can undergo a license change (FOSS to BSL is a very common one) leading to some complicated discussions
There's a standard GitHub uses for license files (which must be at the root of the repo) which fills in the "license" field on the right column of the repo. If the standard isn't met then the link just says "View license". I imagine TabNine is pulling the license from the GitHub API.
https://docs.github.com/en/repositories/managing-your-reposi...
https://github.com/Vedenin/useful-java-links
When master branch is renamed to main, GitHub redirects any old links. https://github.com/github/renaming#renaming-existing-branche...
https://docs.github.com/en/repositories/managing-your-reposi...
https://github.com/Vedenin/useful-java-links
When master branch is renamed to main, GitHub redirects any old links. https://github.com/github/renaming#renaming-existing-branche...
A fine reason for them to bork the useful-java-links case, I guess, but your argument falls over for the 3rd link: https://github.com/MovingBlocks/Terasology/blob/develop/LICE... and the sidebar widget correctly says "Apache-2.0 license"
The code-search application is completely distinct from the IDE-assistant application. They do not share any code, have completely different pipelines for training the backend and completely different datasets. The specific source you mentioned will not be part of the training dataset for Tabnine's IDE assistant. Hope this clarifies.
> The code-search application is completely distinct from the IDE-assistant application
Ok, but how is anyone supposed to know that? There's no verbiage anywhere on those search screens saying that
I thought using the code search with its auto-complete widgets would be a "try it in the browser" version of the product, but what I'm hearing is no, it's just some separate toy trying to be beat Sourcegraph or something?
Ok, but how is anyone supposed to know that? There's no verbiage anywhere on those search screens saying that
I thought using the code search with its auto-complete widgets would be a "try it in the browser" version of the product, but what I'm hearing is no, it's just some separate toy trying to be beat Sourcegraph or something?
Honestly, tab nine is the reason why I didn’t use Copilot for months after I was given initial access — it was just so atrociously bad at what it claimed to help that I was expecting the same from Copilot and thought the concept itself was a broken one. Then out of the blue a coworker mentions he got access to Copilot and loved it, so I gave it a try. Turns out, Copilot just makes things so much easier. I love it. I’m not sure if I’ll pay the subscription right away, but I’m certain I’ll notice the constant annoyance of it missing.
I used TabNine for years, if I recall correctly it was developed by a Waterloo (Ontario - Canada) student and bought by an Israeli company.
Once they changed their EULA my security aware company was like can't use that anymore.
I provided that feedback to them but I guess I was the only one raising eyebrows.
My suggestion is just lower your price to match Copilot, so it's easy on the user deciding between both of you, I am for one sold on how useful Copilot is.
Would you be willing to share your concerns about the EULA? The pricing question is something we are currently discussing as you can imagine. It is surprising that our EULA was a problem but GitHub's was ok given they collect your information for training.
I consider myself pretty advanced user of Tabnine, having purchased the first version from Jacob just when it appeared. I haven't used (and don't plan to) its full-function code completion and only briefly evaluated Copilot. Some thoughts:
- Tabnine's local per-repository model shines best when used on big monolithic repos with lots of similar or boilerplate code. By learning on the same code it's used for it IMO does much better job at detecting local patterns. Also local-model means pretty "safe" from security POV for corporate use (I can't imagine big corps like Google and Amazon allowing use of Copilot for proprietary code).
- Back in they free tier of Tabnine was pretty useless compared to Pro because it severely limited model size. If it's still the case now I hope they have a long evaluation period.
- From usability POV, unfortunately it doesn't provide non-intrusive experience out of the box, which is very annoying and I believe is a reason for many users abandoning it pretty quickly. I had to modify both VSCode extension and VSCode itself (!) so that Tabnine's completion suggestions won't interfere with others.
- Tabnine's local per-repository model shines best when used on big monolithic repos with lots of similar or boilerplate code. By learning on the same code it's used for it IMO does much better job at detecting local patterns. Also local-model means pretty "safe" from security POV for corporate use (I can't imagine big corps like Google and Amazon allowing use of Copilot for proprietary code).
- Back in they free tier of Tabnine was pretty useless compared to Pro because it severely limited model size. If it's still the case now I hope they have a long evaluation period.
- From usability POV, unfortunately it doesn't provide non-intrusive experience out of the box, which is very annoying and I believe is a reason for many users abandoning it pretty quickly. I had to modify both VSCode extension and VSCode itself (!) so that Tabnine's completion suggestions won't interfere with others.
Appreciate the feedback and looks like you are clearly in the developer group that wants very fast small useful completions. Can you share the modifications that you made and perhaps a few more details on what they were?
Both are related to limitations of vscode completion engine, so Tabnine can't really blamed for them in any way.
1. Initially Tabnine's auto-completion was triggered on any character, which best leveraged Tabnine's power but also had inherent problem: when Tabnine was triggered on non-letter character it sometimes prevented Vscode from showing suggestions from other completion sources (LSPs, snippets). There is a discussion in https://github.com/codota/tabnine-vscode/issues/6 with me explaining that the only viable solution is to reduce set of trigger characters to letters only. In the end a fix was pushed that reduced the set of trigger characters, which made the problem less likely but still not solved. The are numerous duplicates of this issue on Github.
2. Another problem is when Vscode has auto-completion suggestions from Tabnine and other sources (LSP, snippets), it frequently puts Tabnine's at the top of list. This is a big no-go for me because most of the time just want to complete the identifier (class field, method etc.). Modifying just the extension code didn't help so in the end I had add a small patch to Vscode itself, which gives lowest score to Tabnine's candidates: https://github.com/sergei-dyshel/vscode/commit/ee73034b9ec6c....
I must admit that both problems can be practically solved by new "inline auto-completion" mechanism in vscode which looks very promising for AI-based completion in general. I'm looking forward to evaluate it.
1. Initially Tabnine's auto-completion was triggered on any character, which best leveraged Tabnine's power but also had inherent problem: when Tabnine was triggered on non-letter character it sometimes prevented Vscode from showing suggestions from other completion sources (LSPs, snippets). There is a discussion in https://github.com/codota/tabnine-vscode/issues/6 with me explaining that the only viable solution is to reduce set of trigger characters to letters only. In the end a fix was pushed that reduced the set of trigger characters, which made the problem less likely but still not solved. The are numerous duplicates of this issue on Github.
2. Another problem is when Vscode has auto-completion suggestions from Tabnine and other sources (LSP, snippets), it frequently puts Tabnine's at the top of list. This is a big no-go for me because most of the time just want to complete the identifier (class field, method etc.). Modifying just the extension code didn't help so in the end I had add a small patch to Vscode itself, which gives lowest score to Tabnine's candidates: https://github.com/sergei-dyshel/vscode/commit/ee73034b9ec6c....
I must admit that both problems can be practically solved by new "inline auto-completion" mechanism in vscode which looks very promising for AI-based completion in general. I'm looking forward to evaluate it.
What's up with this? The title is editorialized as advertisement against a competing product and the content of the link is essentially advertisement too.
It was posted by the VP of Business Development of Tabnine. Is this really organically ending up on the front page or are some bot/vote rings in control?
@dang shouldn't the title at least be changed to "Announcing our Next-generation AI models"?
It was posted by the VP of Business Development of Tabnine. Is this really organically ending up on the front page or are some bot/vote rings in control?
@dang shouldn't the title at least be changed to "Announcing our Next-generation AI models"?
my guess given that I do not have significant karma points is that this is a subject that millions of devs are trying to understand more deeply (a million plus for both Tabnine and Copilot). There are two competitors with significantly different takes on how we solve this problem both technically and securely.
I feel for TabNine.
What they're doing is impactful, impressive - and so eclipsed by an unreasonably well-resourced competitor that they don't even get given much credit.
"Not as good as CoPilot" really doesn't say enough about something that's still very impressive.
Personally I thought it was cool but not game-changing, and moved on when I got into CoPilot. I can't comment on the next-gen models.
What they're doing is impactful, impressive - and so eclipsed by an unreasonably well-resourced competitor that they don't even get given much credit.
"Not as good as CoPilot" really doesn't say enough about something that's still very impressive.
Personally I thought it was cool but not game-changing, and moved on when I got into CoPilot. I can't comment on the next-gen models.
I think it's mostly about having deeper pockets than being clever or original when it comes to A.I. (though these guys just got $15m fwiw).
But this is one thing that worries me about A.I speaking as an indiehacker.
Being scrappy won't soon cut it anymore. Right now you can bootstrap a solid product for $15K, at least a MVP. Soon with the amount of GPU and other resources required to train these models you'll need a min $1M budget and no amount of jugaad will be able to cut it.
But this is one thing that worries me about A.I speaking as an indiehacker.
Being scrappy won't soon cut it anymore. Right now you can bootstrap a solid product for $15K, at least a MVP. Soon with the amount of GPU and other resources required to train these models you'll need a min $1M budget and no amount of jugaad will be able to cut it.
Cost wise the biggest spend is building the large model (CODEX for Copilot). While CODEX is close sourced there are a number of other good open source transformer models from SFDC (T5), AI21, Meta that are the foundational models we use. Going forward we will continue to partner with the community to enable these large models to reach developers in code-time.
Just out of curiosity since you are so knowledgeable in this area.. what would be like a very approx ballpark figure to develop something like codex?
Ballpark guess is that to build CODEX it would be ~$10M.
Has anyone used both Tabnine and Copilot recently? Would like to know if compares to Copilot now.
Last time I checked(end of last year) Tabnine was a far inferior product.
Last time I checked(end of last year) Tabnine was a far inferior product.
Just got done using Tabnine for about 2 months a month ago. Switched to Copilot. There's not much comparison.
I switch between both fairly often because I'm interested in how the space is progressing but tabnine has always been my preference. It seems to aim for a bit of a "lower level" than Copilot which is what I'm looking for.
Assume coPilot proposes a function that can be found in 90% of GPLv3 software. Should all software written by coPilot be GPLv3 as well?
This is a really good question and is tangent to attribution in art-generating packages as well. It's a part of the process that gets ignored because nobody wants to deal with the unethical implications.
If software is doing it, it can't be bad, I guess.
If software is doing it, it can't be bad, I guess.
The problem I had with Tabnine was insane memory consumption to the point where my Mac crawled to a halt. I had to keep killing Tabnine process or just uninstall the plugin. Copilot has been a breeze. I really want to give Tabnine a shot again but I doubt if they can get it to perform as well as Copilot does.
Hi there, Tabnine's DevRel here.
While Copilot serves its code completions through a monolith model, too big to be processed locally (and therefore your code goes into MSFT servers by default), Tabnine's approach is keeping the code it helps writing - private and secured. The way it's done is by using 12 standalone models, which can run locally (but then it'll sometimes consume excessive computing power) OR in encrypted cloud mode, which takes off most of these memory and CPU/GPU resources. You can easily configure the mode you wish to work with through the Tabnine Hub (find the toggle radio button under "Public Model").
I don't when was the last time you've tried coding with Tabnine in the background, but I sure hope that now, with the release of our next-generation abilities you'll give it another go :)
I don't when was the last time you've tried coding with Tabnine in the background, but I sure hope that now, with the release of our next-generation abilities you'll give it another go :)
[deleted]
This is really interesting, $12 a month are a good price IMO.
Is it possible to use TabNine without using the random executable from https://update.tabnine.com/3.2.28/x86_64-unknown-linux-musl/... (Or is there the source available?)
Are there any instructions on to make it work with any other editor (GNOME-Builder in my case)? Is there Vala support?
Edit: Found this: https://github.com/codota/TabNine/blob/master/HowToWriteACli...
Is it possible to use TabNine without using the random executable from https://update.tabnine.com/3.2.28/x86_64-unknown-linux-musl/... (Or is there the source available?)
Are there any instructions on to make it work with any other editor (GNOME-Builder in my case)? Is there Vala support?
Edit: Found this: https://github.com/codota/TabNine/blob/master/HowToWriteACli...
For this specific comparison, it’s essential to start from the technology, as many of the product differences stem from the differences in approach, architecture, and technology choices. Microsoft and OpenAI view AI for software development almost as just another use case for GPT-3, the behemoth language model. Code is text, so they took their language model, fine-tuned it on code, and called the gargantuan 12-billion parameter AI model, Codex.
Copilot’s architecture is monolithic: “one model to rule them all.” It is also completely centralized - only Microsoft can train the model, and only Microsoft can host the model due to the enormous amount of computing resources required for training and inference.
Tabnine, after comprehensively evaluating models of different sizes, favors individualized language models working in concert. Why? Because code prediction is, in fact, a set of distinct sub-problems which doesn't lend itself to the monolithic model approach. For instance: generating the full code of a function in Python based on name and generating the suffix of a line of code in Rust are two problems Tabnine solves well, but the AI model that best fits every such task is different. We found that a combination of specialized models dramatically increases the precision and length of suggestions for our 1M+ users.
A big advantage of Tabnine’s approach is that it can use the right tool for any code prediction task, and for most purposes, our smaller models give great predictions quickly and efficiently. Better yet, most of our models can be run with inexpensive hardware.
Now that we understand the principal difference between Microsoft’s huge monolith and Tabnine’s multitude of smaller models, we can explore the differences between the products:
First, kind of code suggestions. Copilot queries the model relatively infrequently and suggests a snippet or a full line of code. Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose. Similarly, Tabnine Pro also suggests full snippets or lines of code, but since Tabnine also uses smaller and highly efficient AI models, it queries the model while typing. As a user, it means the AI flows with you, even when you deviate from the code it originally suggested.
Second, ability to train the model. Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on. Tabnine can train a private AI model on the specific code from customers’ GitLab/GitHub/BitBucket repositories and thus adjust the suggestions to the project-specific code and infrastructure. Training on customer code is possible because Tabnine is modular, enabling the creation of private customized copies.
Third, Code security and privacy. There are a few aspects of this. Users cannot train or run the Copilot model. The single model is always hosted by Microsoft. Every Copilot user is sending their code to Microsoft; not some of the code, and not obfuscated - all of it. With Tabnine, users can choose where to run the model: on the Tabnine cloud, locally on the developer machine, or on a self-hosted server. This is possible because Tabnine has AI models that can run efficiently with moderate hardware requirements.
In addition, Tabnine makes a firm and unambiguous commitment that no code the user writes is used to train our model. We don’t send to our servers any information about the code that the user writes and the suggestions they’re receiving or accepting.
Fourth, commercial terms. Microsoft currently offers Copilot only as a commercial product for developers, without a free plan (beyond a free trial) or organizational purchase. Tabnine has a great free plan and charges for premium features such as longer code completions and private models trained on customers’ code.
https://tabnine.com/tabnine-vs-github-copilot
Copilot’s architecture is monolithic: “one model to rule them all.” It is also completely centralized - only Microsoft can train the model, and only Microsoft can host the model due to the enormous amount of computing resources required for training and inference.
Tabnine, after comprehensively evaluating models of different sizes, favors individualized language models working in concert. Why? Because code prediction is, in fact, a set of distinct sub-problems which doesn't lend itself to the monolithic model approach. For instance: generating the full code of a function in Python based on name and generating the suffix of a line of code in Rust are two problems Tabnine solves well, but the AI model that best fits every such task is different. We found that a combination of specialized models dramatically increases the precision and length of suggestions for our 1M+ users.
A big advantage of Tabnine’s approach is that it can use the right tool for any code prediction task, and for most purposes, our smaller models give great predictions quickly and efficiently. Better yet, most of our models can be run with inexpensive hardware.
Now that we understand the principal difference between Microsoft’s huge monolith and Tabnine’s multitude of smaller models, we can explore the differences between the products:
First, kind of code suggestions. Copilot queries the model relatively infrequently and suggests a snippet or a full line of code. Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose. Similarly, Tabnine Pro also suggests full snippets or lines of code, but since Tabnine also uses smaller and highly efficient AI models, it queries the model while typing. As a user, it means the AI flows with you, even when you deviate from the code it originally suggested.
Second, ability to train the model. Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on. Tabnine can train a private AI model on the specific code from customers’ GitLab/GitHub/BitBucket repositories and thus adjust the suggestions to the project-specific code and infrastructure. Training on customer code is possible because Tabnine is modular, enabling the creation of private customized copies.
Third, Code security and privacy. There are a few aspects of this. Users cannot train or run the Copilot model. The single model is always hosted by Microsoft. Every Copilot user is sending their code to Microsoft; not some of the code, and not obfuscated - all of it. With Tabnine, users can choose where to run the model: on the Tabnine cloud, locally on the developer machine, or on a self-hosted server. This is possible because Tabnine has AI models that can run efficiently with moderate hardware requirements.
In addition, Tabnine makes a firm and unambiguous commitment that no code the user writes is used to train our model. We don’t send to our servers any information about the code that the user writes and the suggestions they’re receiving or accepting.
Fourth, commercial terms. Microsoft currently offers Copilot only as a commercial product for developers, without a free plan (beyond a free trial) or organizational purchase. Tabnine has a great free plan and charges for premium features such as longer code completions and private models trained on customers’ code.
https://tabnine.com/tabnine-vs-github-copilot
Might be worth disclosing you are the “VP (of) Ecosystem and Business Development” for Tabnine in any comments that your pitching Tabnine; while you’re at it, might not hurt to add that to your HN profile.
The other commenters have pointed out that this is basically just PR, but even if we ignore that entirely, it doesn't make a lot of sense. A few issues I pulled out somewhat at random:
> Copilot queries the model relatively infrequently
Huh? It queries whenever you stop typing.
> Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose
This is at best false, at worse disingenuous. Yes, it won't insert a completion in the middle of a line, but this is trivially solved by inserting a newline in the middle of the line and then triggering a completion at the end of the line you just created.
> In addition, Tabnine makes a firm and unambiguous commitment that no code the user writes is used to train our model.
Copilot has this option too...
> Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on.
I mean, anyone who has used Copilot for more than 5 seconds would know that this isn't true. Copilot does a fantastic job at reading the current file and providing suggestions relevant to the file, rather than some hypothetical "average of GitHub"
> Copilot queries the model relatively infrequently
Huh? It queries whenever you stop typing.
> Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose
This is at best false, at worse disingenuous. Yes, it won't insert a completion in the middle of a line, but this is trivially solved by inserting a newline in the middle of the line and then triggering a completion at the end of the line you just created.
> In addition, Tabnine makes a firm and unambiguous commitment that no code the user writes is used to train our model.
Copilot has this option too...
> Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on.
I mean, anyone who has used Copilot for more than 5 seconds would know that this isn't true. Copilot does a fantastic job at reading the current file and providing suggestions relevant to the file, rather than some hypothetical "average of GitHub"
> Huh? It queries whenever you stop typing.
That's relatively infrequently - TabNine offered me accurate completion while still writing my code, whereas with Copilot I not only have to wait for it to return the answer, I also have to hope it knows an answer. If it doesn't, too bad, lost time for nothing.
That's relatively infrequently - TabNine offered me accurate completion while still writing my code, whereas with Copilot I not only have to wait for it to return the answer, I also have to hope it knows an answer. If it doesn't, too bad, lost time for nothing.
Thank you for laying out facts for us, at least from Tabnine perspective.
I didn't use Tabnine but anecdotally heard it is not as good to guess whole methods the way Copilot is. Now I understand why.
The only suggestion I would make, maybe lower the price, MS is bigger player, they pretty much set the standard now, if they have lower price even symbolically, it would hurt you more.
I personally am happy with Copilot, had some really wonderful moments and savings in time searching for proper method syntax is where I get a lot of benefit. I should spend time to get to know Tabnine as well so not to be ignorant on this new technology.
I didn't use Tabnine but anecdotally heard it is not as good to guess whole methods the way Copilot is. Now I understand why.
The only suggestion I would make, maybe lower the price, MS is bigger player, they pretty much set the standard now, if they have lower price even symbolically, it would hurt you more.
I personally am happy with Copilot, had some really wonderful moments and savings in time searching for proper method syntax is where I get a lot of benefit. I should spend time to get to know Tabnine as well so not to be ignorant on this new technology.
I noticed there is a free tier for individual users with limited capability. But I agree with the thrust of your argument here.
Good feedback and thank you for taking the time to share
There are some things in your comment that are wrong, or don't quite follow.
> Copilot’s architecture is monolithic: “one model to rule them all.” It is also completely centralized - only Microsoft can train the model
It's true that only Microsoft can train the model—for now. But GPT-3 is a monolithic model, and OpenAI allows fine-tuning. There's nothing about the architecture that prohibits Microsoft from offering customers the ability to fine-tune the model.
> Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose.
Here's a screenshot of GitHub Copilot suggesting code in the middle of a line:
https://imgur.com/gallery/Jp2qg7X
> Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on.
If the model is big enough that seems to be an advantage, not a disadvantage. At least, when I use GPT-3 to create a 4chan greentext, or an X-Files script, or a presidential speech, or an article by Hunter S. Thompson (or a presidential speech by Hunter S. Thompson) it doesn't particularly feel like it's averaging things out.
> Copilot’s architecture is monolithic: “one model to rule them all.” It is also completely centralized - only Microsoft can train the model
It's true that only Microsoft can train the model—for now. But GPT-3 is a monolithic model, and OpenAI allows fine-tuning. There's nothing about the architecture that prohibits Microsoft from offering customers the ability to fine-tune the model.
> Copilot does not suggest code in the middle of the line, as its AI model is not best suited for this purpose.
Here's a screenshot of GitHub Copilot suggesting code in the middle of a line:
https://imgur.com/gallery/Jp2qg7X
> Copilot uses one universal AI model, which means that every user is getting the same generic assistance based on an “average of GitHub”, regardless of the project they're working on.
If the model is big enough that seems to be an advantage, not a disadvantage. At least, when I use GPT-3 to create a 4chan greentext, or an X-Files script, or a presidential speech, or an article by Hunter S. Thompson (or a presidential speech by Hunter S. Thompson) it doesn't particularly feel like it's averaging things out.
I fail to see how in the screenshot you provided Copilot suggests code _in the middle_ of the line.
What I see is Copilot suggesting an entire line. Or does the suggestion start on the Err line with the "{" ?
What I see is Copilot suggesting an entire line. Or does the suggestion start on the Err line with the "{" ?
On the highlighted line I typed `panic!("Error while ` and Copilot suggested finishing the line I started with `converting GeoJSON geometry to geometry: {}", e);`
Complete marketing PR. Probably you should be declaring where do you work when posting PR comments
To be fair. Tabnine has an actual free plan for personal use. The GitHub Copilot one is only for open source maintainers and educational institutions.
All though in the past I found the limited suggestions offered with the Tabnine free plan too limiting to be of much help.
All though in the past I found the limited suggestions offered with the Tabnine free plan too limiting to be of much help.
It’s good to see more players in this space.
Tabnine was around at least one year longer than Copilot though
Thanks for clarifying. I wasn’t aware of it back then unfortunately.
"Understanding AI for coders..."
For any coders who might think this article will help them understand AI, the actual title is:
Announcing our Next-generation AI models
Perhaps it has changed since being posted but it's just a plug.
For any coders who might think this article will help them understand AI, the actual title is:
Announcing our Next-generation AI models
Perhaps it has changed since being posted but it's just a plug.
I used the free version of Tabnine and beta tested Copilot.
I actually really liked Tabnine but sometimes the tons of suggestions got annoying. I upgraded my IDE and forgot to add copilot but didn't feel like I needed to. Maybe that's a mistake... but sometimes it felt like it was getting in the way and I did encounter a few times it would create "bugs" which I would have to spend time going back and fixing.
Not trying to knock these though. They're both very impressive tools.
I actually really liked Tabnine but sometimes the tons of suggestions got annoying. I upgraded my IDE and forgot to add copilot but didn't feel like I needed to. Maybe that's a mistake... but sometimes it felt like it was getting in the way and I did encounter a few times it would create "bugs" which I would have to spend time going back and fixing.
Not trying to knock these though. They're both very impressive tools.
I've been using TabNine for years and, at least with Python, the suggestions are superb, and often uncanny - it occasionally suggests the variable name I was thinking of using before it's appeared anywhere in the code. I love TabNine dearly, but the Sublime Text plugin for it has basically been left to rot which is really sad. It doesn't integrate at all well with ST4.
Maybe tabnine changed some things with their new Pro version included multiline completions
https://t.co/YGfmAZopwe
https://t.co/YGfmAZopwe
I use Tabnine completion in JetBrains CLion when I write Rust code. I’ve found it to be quite useful. Haven’t tried GitHub Copilot yet, so don’t know how Copilot compares to it.
I used Tabnine before GitHub Copilot and I never looked back. It feels a bit like upgrading from GPT2 to GPT3.
I mean, that's literally what happened. At least initially Tabnine was based on the GPT2 model trained on code. Then GitHub launched Copilot using the OpenAI Codex model which is based on GPT3. I guess this is why several people have commented on the marked improvement when adopting GitHub Copilot.
I have no idea how Tabnine builds their models today, and how they perform compared to Copilot. I guess one advantage could be lower latency in suggestions that they claim come from training smaller more specific models. But the way I find Copilot working for me is that it thinks about the same time I need to think and then it makes a suggestion for a good chunk of code. If my thinking and Copilot's thinking match up then I can save myself a good bit of typing.
I have no idea how Tabnine builds their models today, and how they perform compared to Copilot. I guess one advantage could be lower latency in suggestions that they claim come from training smaller more specific models. But the way I find Copilot working for me is that it thinks about the same time I need to think and then it makes a suggestion for a good chunk of code. If my thinking and Copilot's thinking match up then I can save myself a good bit of typing.
There's a decent comparison outlining the differences by Tabnine's CEO here (disclaimer - I work there).
https://twitter.com/drorwe/status/1539293063117516801
Tabnine and Copilot feel like the endboss for various plagiarism checkers. I'm curious how universities will adapt to this.
How have universities adapted to copy-pasting code off of StackOverflow?
Or rewriting code from books?
Ouch $17 AUD/month for a single user!
And for a dead product. These guys need to up their game fast, pivot, or fall off the radar.
Can we hope for a local, open-source version of these models some day?
it is a question we are always asking but not sure what people would do if they were open-sourced. Did you have a specific use case in mind?
Very editorialized title.
[deleted]
I use Copilot for its multi-line code generation. It's vastly superior. (Sorry Tabnine). Gotta give credit to Tabnine for kicking off this space -- they had a multi-year headstart on Copilot and it was pretty cool when it first launched, but it just doesn't come close to Copilot. OpenAI is kicking butt here.
Having said that, I still find value out of Tabnine -- I keep it installed it powers the pop-up autocomplete (eg Intellisense replacement). The Tabnine suggestions are immediate (as opposed to Copilot which takes a few seconds), and in a lot of cases it's super helpful for completing a variable name after 1 character, etc.
If I ESC out of the Tabnine suggestion, then Copilot takes over and a few seconds later I get a full line/multi-line suggestion.
I dunno - it's been working really well for me. Peaceful coexistence :)