Gitlab’s AI-assisted code suggestions(about.gitlab.com)
about.gitlab.com
Gitlab’s AI-assisted code suggestions
https://about.gitlab.com/solutions/code-suggestions/
159 comments
Did you change the way you worked to help facilitate it at all?
I get great results from Copilot, but that's because I do a bunch of things to help it work for me.
One example: I'll often paste in a big chunk of text - the class definition for an ORM model for example - then use Copilot to write code that uses that class, then delete the code I pasted in again later.
Or I'll add a few lines of comments describing what I'm about to do, and let Copilot write that code for me.
Did you try the trick where you paste in a bunch of code and then start writing tests for it and Copilot spits out the test code for you?
A few more notes about how I've used Copilot here:
- https://til.simonwillison.net/gpt3/writing-test-with-copilot
- https://til.simonwillison.net/gpt3/reformatting-text-with-co...
I've also used Copilot to take educated guesses at things - like an inline mini-ChatGPT - with good results:
- https://til.simonwillison.net/gpt3/guessing-amazon-urls
As with so many other AI tools, Copilot is desperately lacking detailed documentation. It's not at all obvious how to get the most out of it.
I get great results from Copilot, but that's because I do a bunch of things to help it work for me.
One example: I'll often paste in a big chunk of text - the class definition for an ORM model for example - then use Copilot to write code that uses that class, then delete the code I pasted in again later.
Or I'll add a few lines of comments describing what I'm about to do, and let Copilot write that code for me.
Did you try the trick where you paste in a bunch of code and then start writing tests for it and Copilot spits out the test code for you?
A few more notes about how I've used Copilot here:
- https://til.simonwillison.net/gpt3/writing-test-with-copilot
- https://til.simonwillison.net/gpt3/reformatting-text-with-co...
I've also used Copilot to take educated guesses at things - like an inline mini-ChatGPT - with good results:
- https://til.simonwillison.net/gpt3/guessing-amazon-urls
As with so many other AI tools, Copilot is desperately lacking detailed documentation. It's not at all obvious how to get the most out of it.
I sometimes prompt Copilot with if statements and var declarations and it actually ends up suggesting missed corner cases and fixing bugs. Definitely, there's an art and form to using it.
I wonder if this quest for the perfect prompt is dumbing me down, though.
I wonder if this quest for the perfect prompt is dumbing me down, though.
Using Copilot has taught me how to write better code comments. And the improved comment quality benefits everyone, not just the AI.
Even something as simple as adding a doc block for a method can boost the quality quite a bit.
I could live without it, but I'd also keep paying if the price increased.
Even something as simple as adding a doc block for a method can boost the quality quite a bit.
I could live without it, but I'd also keep paying if the price increased.
At the current price, if it saves me a single hour in the course of a year it pays for itself. And it definitely does.
I agree. It has made me better at articulating what I want to do, what a function does, and something as granular as what a particular loop hopes to achieve
Just today Copilot saved me close to 1 hour of typing and replaced it essentially with ~1 minute of pressing Tab.
I was hooking IAudioClient COM class to capture and silence arbitrary app's audio, and as soon as I wrote signatures for its members Copilot was able to generate skeletal stub implementation with logging as well as hooking code totaling about 150 lines of Rust.
I was hooking IAudioClient COM class to capture and silence arbitrary app's audio, and as soon as I wrote signatures for its members Copilot was able to generate skeletal stub implementation with logging as well as hooking code totaling about 150 lines of Rust.
This is an example of why codebases with strict typing & clear conventions will benefit the most from copilot.
It would be even better if Copilot was able to make use of the strict typing to prevent generating code that doesn't typecheck in the first place, to be honest. Copilot as-is actually feels more useful for dynamic languages, because static languages get really far without any LLMs and simple static autocomplete gets you really far. So there's more ground to make up.
In addition, Copilot benefits from a large training set, which I guess is best for Python and JavaScript as two very popular languages. Strictly typed languages seem like they are generally less well represented, at least among public GitHub projects.
In addition, Copilot benefits from a large training set, which I guess is best for Python and JavaScript as two very popular languages. Strictly typed languages seem like they are generally less well represented, at least among public GitHub projects.
To offer a counter opinion: while I can live without ChatGPT and cancelled my OpenAI plus membership, I cannot live without Copilot and would be willing to pay a lot to keep it.
There are a lot of tricks to optimizing output from Copilot ("Copilot-Driven Development"), I wrote a little about what I've discovered here:
https://gavinray97.github.io/blog/a-day-without-a-copilot#co...
There are a lot of tricks to optimizing output from Copilot ("Copilot-Driven Development"), I wrote a little about what I've discovered here:
https://gavinray97.github.io/blog/a-day-without-a-copilot#co...
> Use explicit names, erring on the side of overly-verbose (e.g. ensureUserIsLoggedIn() instead of euil())
I don't care that much if I sound pretentious or too harsh, but how is this not common sense? Maybe the general standards in the industry are way lower than they used to be. This looks like when one of the consultants hired by our company and paid 10k+ per month for a ISO 27001 certification said: you need to make sure your password fields show a * instead of the actual character.
After using copilot for 2 months, I found it somehow useful for very generic repetitive cases, but for complex specific stuff it was very bad. I'd rather spend an hour writing code and understanding the problem I'm trying to solve than spending an hour understanding and adapting some generic code that partially does what it needs to do. And if there are people out there that fully trust it and then deploy code that they don't understand and can't explain themselves, then we're in for a lot of fun in the future.
I don't care that much if I sound pretentious or too harsh, but how is this not common sense? Maybe the general standards in the industry are way lower than they used to be. This looks like when one of the consultants hired by our company and paid 10k+ per month for a ISO 27001 certification said: you need to make sure your password fields show a * instead of the actual character.
After using copilot for 2 months, I found it somehow useful for very generic repetitive cases, but for complex specific stuff it was very bad. I'd rather spend an hour writing code and understanding the problem I'm trying to solve than spending an hour understanding and adapting some generic code that partially does what it needs to do. And if there are people out there that fully trust it and then deploy code that they don't understand and can't explain themselves, then we're in for a lot of fun in the future.
Same. It’s funny reading comments on here. Even gpt4 I think is just ok. Maybe the people who see these huge increases are more junior?
I'm definitely not more junior, and I'm seeing massive productivity improvement from Copilot and GPT-4 - but I'm finding it takes a lot of expertise to get the best results, both in my ability as a programmer and in terms of knowing the best ways to use the AI tools.
Learning how to get the best results of them takes a great deal of experimentation.
Learning how to get the best results of them takes a great deal of experimentation.
This has been my experience as well. I actually avoid recommending Copilot to juniors because I think using it requires a deeper understanding of what you're trying to do than is required to just write the code by hand. When you have that understanding it can be a huge time saver, but it's not something you can just pick up and magically ask to solve your problems for you.
I’m curious what your workflow is like switching between copilot and gpt-4. I typically have an open window for each, though this can feel more cumbersome than necessary at times.
I actually use the ChatGPT UI more than copilot - I write code with it while I'm out walking the dog, by describing what I want, then copy and paste it into my text editor on my laptop when I get home.
I'm increasingly using my LLM CLI tool for quick lookups, and sometimes to make changes to existing code too - see https://github.com/simonw/symbex/releases/tag/1.0
I'm increasingly using my LLM CLI tool for quick lookups, and sometimes to make changes to existing code too - see https://github.com/simonw/symbex/releases/tag/1.0
Copilot Chat is pretty great. Highly recommend signing up for the waitlist.
https://github.com/github-copilot/chat_waitlist_signup/join
https://github.com/github-copilot/chat_waitlist_signup/join
>i usually have an open window for each
Do you not use the direct vscode integration? I don't think I'd use it otherwise. It's been hugely convenient with.
Do you not use the direct vscode integration? I don't think I'd use it otherwise. It's been hugely convenient with.
I've been working as a programmer for almost 15 years and I work on fairly touchy (perf, reliability-wise) networked systems in a programming language with a good type system. I find Copilot to be very valuable even if I have to give it a little bump or edit what it produces sometimes.
My experience was very similar. Copilot became a sort of improved autocomplete, that would save copy/pasting repeated variable names into tests, logs, etc, but offered no other significant improvements. Its suggestions were frequently incorrect, often not obviously so.
> that would save copy/pasting repeated variable names into tests, logs, etc
This is all it takes for me to get value out of my subscription. It's saving me time by anticipating my next test / block of code based on what I've previously written, and that saved time really adds up.
Sure it's not "Jesus take the wheel" level AI yet, but time is money, and so is my sanity.
This is all it takes for me to get value out of my subscription. It's saving me time by anticipating my next test / block of code based on what I've previously written, and that saved time really adds up.
Sure it's not "Jesus take the wheel" level AI yet, but time is money, and so is my sanity.
I find it’s really similar to having a team of junior devs under you. Lots of output, generally good for problems that are googleable, need heavy guidance if the problem is both new and niche, lots of subtle bugs.
The first two points in @gavinray’s methodology being unfortunately critical.
The first two points in @gavinray’s methodology being unfortunately critical.
This month, for the third time, I evaluated copilot. It has vastly improved. So far that this time I'll probably pay for it.
I think it's biggest benefit isn't for me, though. Copilot is great at churning out predictable or repetitive stuff, much less in writing a abstractions over or around that repetitive or predictable stuff. E.g. it's great at writing yet another CreditcardPayment ActiveRecord model for Rails. But less so for an obscure business domain model used outside of a framework. And great at writing fifteen getters and setters, less for an abstract base class, DSL or macro that introduces all these getters and setters for me.
It's also bad at certain rewrites. A codebase was slowly rewritten from OldStyle (and old libs) to NewStyle. It stubbornly kept suggesting OldStyle snippets.
And last, I find Copilot has a far higher ROI in some languages and frameworks than others. E.g. dynamic languages (like Ruby) have very poor LSP and intellisense support compared to typed and static (like Rust). So the added benefit differs a lot based on where it's used, is my experience.
I guess esp. the latter is why I too am underwhelmed. But also why I'll keep using it, this time, for when I do the inevitable Rails, JavaScript or Panda's gigs. But less so for my weird eventsourced, hexagonal, Rust project.
I think it's biggest benefit isn't for me, though. Copilot is great at churning out predictable or repetitive stuff, much less in writing a abstractions over or around that repetitive or predictable stuff. E.g. it's great at writing yet another CreditcardPayment ActiveRecord model for Rails. But less so for an obscure business domain model used outside of a framework. And great at writing fifteen getters and setters, less for an abstract base class, DSL or macro that introduces all these getters and setters for me.
It's also bad at certain rewrites. A codebase was slowly rewritten from OldStyle (and old libs) to NewStyle. It stubbornly kept suggesting OldStyle snippets.
And last, I find Copilot has a far higher ROI in some languages and frameworks than others. E.g. dynamic languages (like Ruby) have very poor LSP and intellisense support compared to typed and static (like Rust). So the added benefit differs a lot based on where it's used, is my experience.
I guess esp. the latter is why I too am underwhelmed. But also why I'll keep using it, this time, for when I do the inevitable Rails, JavaScript or Panda's gigs. But less so for my weird eventsourced, hexagonal, Rust project.
I also find Copilot to be rather underwhelming and half the time I find myself wishing it didn't go as overboard with the suggestions as it does. Sometimes it suggests exactly what I want to type and then adds a bunch of extra crap towards the end so it is the same amount of effort to type it manually or accept the suggestion and erase what I don't need.
Also its inability to match parens and braces/brackets is legendary. So annoying.
But! It is extremely helpful for filling in error messages. I don't think I wrote a `throw new ` line fully since I started using it. The quality and information density of my error messages has increased considerably.
Not sure if it is worth keeping for just that but it is a nice benefit.
Also its inability to match parens and braces/brackets is legendary. So annoying.
But! It is extremely helpful for filling in error messages. I don't think I wrote a `throw new ` line fully since I started using it. The quality and information density of my error messages has increased considerably.
Not sure if it is worth keeping for just that but it is a nice benefit.
I found the same, and I was really hoping for a productivity boost. I really wonder how much this correlates to developer speed. I code using Vim and type probably 80-100wpm. Typing is usually not the bottleneck for me. Thinking is. Copilot is good at typing, but not good at thinking. IME. Occasionally it will surprise me with a valid contribution, which I then need to check just as carefully as if I were writing it myself.
TBH, using this as a fancy smart autocomplete doesn't seem like the best idea. I can't believe that I'm suggesting something that sounds like Clippy... but...
It really needs to be something where you can rattle off a list of requirements and have it build the code for you. The code context is necessary, but not sufficient.
It really needs to be something where you can rattle off a list of requirements and have it build the code for you. The code context is necessary, but not sufficient.
It’s coming up: https://github.com/features/preview/copilot-x
I don't get the thing with automated PR and commit messages. They mostly describe _what_ was done and not why.
Also, if you enhance your available information with AI, why not just write no message at all (or a short one as always) and use the AI when _looking_ at the commits/PRs? The AI will certainly be better at this in a later stage because they themselves will get better and they will have more context due to being able to look at the commits that followed the PR/commit.
There is no point in generating the commit message when committing.
When auto-generated messages will become a commodity, there will be issues with it. For example, if the message doesn't fit with the actual commit contents (semantically or syntactically), I have to think hard whether this is an AI-message-generator-bug, or the author missed something or wether I am missing something. This is not cool and makes reading commit messages harder.
Also, if you enhance your available information with AI, why not just write no message at all (or a short one as always) and use the AI when _looking_ at the commits/PRs? The AI will certainly be better at this in a later stage because they themselves will get better and they will have more context due to being able to look at the commits that followed the PR/commit.
There is no point in generating the commit message when committing.
When auto-generated messages will become a commodity, there will be issues with it. For example, if the message doesn't fit with the actual commit contents (semantically or syntactically), I have to think hard whether this is an AI-message-generator-bug, or the author missed something or wether I am missing something. This is not cool and makes reading commit messages harder.
I'm excited to continue to be disappointed.
Yeah not worth it at the moment. Often it annoys me be overwriting legitimate suggestions from the editor.
Only reason I still have it is because the beta claims to have GPT-4 integration and that could prove to be a very powerful feature. But I still haven;t gotten an invite to install.
Only reason I still have it is because the beta claims to have GPT-4 integration and that could prove to be a very powerful feature. But I still haven;t gotten an invite to install.
Are there other people who find that IntelliJ's autocomplete suggestions and shortcuts (including live templates&Co) already do 80% of what Copilot is trying?
Yea, but that 20% is ultra powerful. It handles the fuzzy things really well and it pattern matches even better.
If I write a comment describing what I’m doing, it will generate multiple lines of correct code
If I write a comment describing what I’m doing, it will generate multiple lines of correct code
People love to cargo-cult. It's wild. It was okay but I'm happier with ChatGPT for one-off things I don't know how to do "give me a sql query that does this obscure thing" etc..
It's okay that it didn't fit well into your workflow, but accusing those who use it of cargo culting isn't constructive. Other people don't have the same workflow as you do, and as simonw noted above, some of us actually adjusted our workflow to get the most out of Copilot.
Not everyone who used it cargo-culted but certainly some did.
I get that gitlab needs more features to continue to sell.
But honestly for me it reduces the attractiveness of gitlab. Just adding half working features does not increase the value for me.
Please finish and polish the existing Features before adding hype feature 93939. Maybe add a policy that your open feature and big count should be less than 20K or something.
Examples: Security scanning has a god awful UI and UX.
The whole project management stuff is barely usable
Pipelines have loads of weird half features and missing QOL. E G. Adding saeif support instead of just your own warnings format.
But honestly for me it reduces the attractiveness of gitlab. Just adding half working features does not increase the value for me.
Please finish and polish the existing Features before adding hype feature 93939. Maybe add a policy that your open feature and big count should be less than 20K or something.
Examples: Security scanning has a god awful UI and UX.
The whole project management stuff is barely usable
Pipelines have loads of weird half features and missing QOL. E G. Adding saeif support instead of just your own warnings format.
It's honestly doubly annoying when there's multiple times per month where I run into some issue / straight up bug which I then find an ongoing bug ticket from like 2018 for, that's still open but apparently not being worked on.
I really like Gitlab, I think the CI experience is a lot better than, for instance, Github Actions, but I also don't ever see me using a good chunk of the stuff that's been shoveled on in the last few years and I don't really know anyone that does.
I really like Gitlab, I think the CI experience is a lot better than, for instance, Github Actions, but I also don't ever see me using a good chunk of the stuff that's been shoveled on in the last few years and I don't really know anyone that does.
At least for python and dotnet, I think github actions are superior.
Gitlab did not support poetry for ages there where bugs with python galore and for dotnet they do not speak the same language - e.g. issues in dotnet are reported as saeif, but gl has it own weird format.
And yes, you are totally right with the bug tracker. It's often funny how long the ticket is just because it was retagged sooo many times.
Gitlab did not support poetry for ages there where bugs with python galore and for dotnet they do not speak the same language - e.g. issues in dotnet are reported as saeif, but gl has it own weird format.
And yes, you are totally right with the bug tracker. It's often funny how long the ticket is just because it was retagged sooo many times.
Crucial: do they train on GPL code? If I am to use this tool, I must abide by the license terms of the training data. Even if it is found that the GPL does not cover LLM responses as derivative works, I would prefer to be on the safe side and refuse to use models trained on software with non-permissive licenses unless I am building Free Software.
Even MIT licensed code requires you to preserve the copyright and permission notice.
If a human did what these language models are doing (output derivative works with the copyright and license stripped), it would be a license violation. When humans want to create a new implementation with clean IP, they have one team study the IP-encumbered code and write a spec, then a different team writes a new implementation according to the spec. LM developers could have similar practices, with separately-trained components that create an auditable intermediate representation and independently create new code based on that representation. The tech isn't up to that task and the LM authors think they're going to get away with laundering what would be plagiarism if a human did it.
If a human did what these language models are doing (output derivative works with the copyright and license stripped), it would be a license violation. When humans want to create a new implementation with clean IP, they have one team study the IP-encumbered code and write a spec, then a different team writes a new implementation according to the spec. LM developers could have similar practices, with separately-trained components that create an auditable intermediate representation and independently create new code based on that representation. The tech isn't up to that task and the LM authors think they're going to get away with laundering what would be plagiarism if a human did it.
Why can't AI do the same: copyrighted code -> spec -> generated code.
... and then execute copyrighted code -> trace resulting values -> tests for new code.
AI could do clean room reimplementation of any code to beef up the training set. It can also make sure the new code is different from the old code at ngram-level, so even by chance it should not look the same.
Would that hold up in court? Is it copyright laundering?
... and then execute copyrighted code -> trace resulting values -> tests for new code.
AI could do clean room reimplementation of any code to beef up the training set. It can also make sure the new code is different from the old code at ngram-level, so even by chance it should not look the same.
Would that hold up in court? Is it copyright laundering?
Language models don't understand anything, they just manipulate tokens. It is a much harder task to write a spec (that humans and courts can review if needed to determine is not infringement) and (with a separately trained tool) implement the spec. The tech just isn't ready and it's not clear that language models will ever get there.
What language models could do easily is to obfuscate better so the license violation is harder to prove. That's behavior laundering -- no amount of human obfuscation (e.g., synonym substitution, renaming variables, swapping out control structures) can turn a plagiarized work into one that isn't. If we (via regulators and courts) let the Altmans of the world pull their stunt, they're going to end up with a government-protected monopoly on plagiarism-laundering.
What language models could do easily is to obfuscate better so the license violation is harder to prove. That's behavior laundering -- no amount of human obfuscation (e.g., synonym substitution, renaming variables, swapping out control structures) can turn a plagiarized work into one that isn't. If we (via regulators and courts) let the Altmans of the world pull their stunt, they're going to end up with a government-protected monopoly on plagiarism-laundering.
Isn’t the language model itself the spec?
Potentially for all of the inputs at once.
Potentially for all of the inputs at once.
> When humans want to create a new implementation with clean IP, they have one team study the IP-encumbered code and write a spec, then a different team writes a new implementation according to the spec.
Maybe at a FAANG or some other MegaCorp, but most companies around barely have a single dev team at all, or if they're larger barely have one per project.
Maybe at a FAANG or some other MegaCorp, but most companies around barely have a single dev team at all, or if they're larger barely have one per project.
Has anyone been able to create a prompt that GPT4 replies to with copyrighted content (or content extremely similar to the original content)?
I'm curious how easy or difficult it is to get GPT to spit out content (code or text) that could be considered obvious infringement.
Tempted to give it half of some closed-source or restrictive licensed code to see if it auto-completes the other half in a manner that is obviously recreating the original work.
I'm curious how easy or difficult it is to get GPT to spit out content (code or text) that could be considered obvious infringement.
Tempted to give it half of some closed-source or restrictive licensed code to see if it auto-completes the other half in a manner that is obviously recreating the original work.
I don't know about GPT-4 but you could get ChatGPT to spit Carmac's Fast Inverse square root with the comments and all (I can't find the tweet though…)
Edit: it wasn't ChatGPT but Copilot see https://twitter.com/mitsuhiko/status/1410886329924194309
Edit: it wasn't ChatGPT but Copilot see https://twitter.com/mitsuhiko/status/1410886329924194309
I can reproduce when prompted all the lyrics to Bohemian Rhapsody, but my doing so isn’t automatically copyright infringement. It would depend on where, when, how, in front of what audience, and to what purpose I was reciting them as to whether it was irrelevant to copyright law, protected under some copyright use case, civilly infringing, or criminally infringing copyright abuse.
The same applies to GPT. It could reproduce Bohemian Rhapsody lyrics in the course of answering questions and there’s no automatic breach of copyright that’s taking place. It’s okay for GPT to know how a well known song goes.
If copilot ‘knows how some code goes’ and is able to complete it, how is that any different?
The same applies to GPT. It could reproduce Bohemian Rhapsody lyrics in the course of answering questions and there’s no automatic breach of copyright that’s taking place. It’s okay for GPT to know how a well known song goes.
If copilot ‘knows how some code goes’ and is able to complete it, how is that any different?
OK, it can exist without breaking any laws, but if you can't release anything it helps you write, what's the point?
There’s a clear separation between the training process which looks at code and outputs nothing but weights, and the generation process which takes in weights and prompts and produces code.
The weights are an intermediate representation that contains nothing resembling the original code.
The weights are an intermediate representation that contains nothing resembling the original code.
But the original content is frequently recoverable.
You can't just take copyrighted code, base 64 it, sent it to someone, have them decode it, and claim there was no copyright violation.
From my (admittedly vague) understanding copyright law cares about the lineage of data, and I don't see how any reasonable interpretation could consider that the lineage doesn't pass through models.
IANAL
You can't just take copyrighted code, base 64 it, sent it to someone, have them decode it, and claim there was no copyright violation.
From my (admittedly vague) understanding copyright law cares about the lineage of data, and I don't see how any reasonable interpretation could consider that the lineage doesn't pass through models.
IANAL
> But the original content is frequently recoverable.
What if we train the model on paraphrases of the copyrighted code? The model can't reproduce exactly what it has not seen.
Also consider the size ratio - 1TB of code+text ends up into 1GB of model weights. There is no space to "memorize" the training set, it can only learn basic principles and how to combine them to generate code on demand.
The copyright law in principle should only protect expression, not ideas. As long as the model learns the underlying principles without copying the superficial form, it should be ok. That's my 2c
What if we train the model on paraphrases of the copyrighted code? The model can't reproduce exactly what it has not seen.
Also consider the size ratio - 1TB of code+text ends up into 1GB of model weights. There is no space to "memorize" the training set, it can only learn basic principles and how to combine them to generate code on demand.
The copyright law in principle should only protect expression, not ideas. As long as the model learns the underlying principles without copying the superficial form, it should be ok. That's my 2c
The fact that this is a problem is a bug in copyright law, not a shortcoming of the LLM.
> The weights are an intermediate representation that contains nothing resembling the original code.
So is the ELF.
So is the ELF.
The neurons in my brain when I plagiarize are just arrangements of atoms that contain nothing that resembles orginal code/text passages/etc.
The trained weights of a GPT model are a frozen, static, transmissible representation. They’re not equivalent to the live state of a brain.
Pretty equivalent to the snapshot of a live brain. Those inside it are even called neurons and neural network
No, they are the weights that are used to configure a neural network. They’re a map of how to build a useful brain, not a neural state.
Machine learning neural networks have almost nothing to do with how brains work besides a tenuous mathematical relation that was conceived in the 1950s.
You can say that if you want to nitpick, but there are recent studies showing that neural and brain representations align rather well, to the point that we can predict what someone is seeing from brain waves, or generate the image with stable diffusion.
https://sites.google.com/view/stablediffusion-with-brain/
I think brain to neural net alignment is justified by the fact that both are the result of the same language evolutionary process. We're not all that different from AIs, we just have better tools and environments, and evolutionary adaptation for some tasks.
Language is an evolutionary system, ideas are self replicators, they evolve parallel to humans. We depend on the accumulation of ideas, starting from scratch would be hard even for humans. A human alone with no language resources of any kind would be worse than a primitive.
The real source of intelligence is the language data from which both humans and AIs learn, model architecture is not very important. Two different people, with different neural wiring in the brain, or two different models, like GPT and T5 can learn the same task given the training set. What matters is the training data. It should be credited with the skills we and AIs obtain. Most of us live our whole lives at this level and never come up with an original idea, we're applying language to tasks like GPT.
https://sites.google.com/view/stablediffusion-with-brain/
I think brain to neural net alignment is justified by the fact that both are the result of the same language evolutionary process. We're not all that different from AIs, we just have better tools and environments, and evolutionary adaptation for some tasks.
Language is an evolutionary system, ideas are self replicators, they evolve parallel to humans. We depend on the accumulation of ideas, starting from scratch would be hard even for humans. A human alone with no language resources of any kind would be worse than a primitive.
The real source of intelligence is the language data from which both humans and AIs learn, model architecture is not very important. Two different people, with different neural wiring in the brain, or two different models, like GPT and T5 can learn the same task given the training set. What matters is the training data. It should be credited with the skills we and AIs obtain. Most of us live our whole lives at this level and never come up with an original idea, we're applying language to tasks like GPT.
I think this view is incredibly dangerous to any kind of skills mastery. It has the potential to completely destroy the knowledge economy and eventually degrade AI due to a dearth of training data.
It reminds me of people needing to do a "clean room implementation" without ever seeing similar code. I feel like a human being who read a bunch of code and then wrote something similar without copy/paste or looking at the training data should be protected, and therefore an AI should too.
Okay, that’s an argument from consequences, but is the view factually wrong?
I mean those consequences are why patent law exists. New technology may require new regulatory frameworks, like we've been doing since railroads. The idea that we could not amend law and that we need to pedantically say "well this isn't illegal now" as an excuse for doing something unethical and harmful to the economy is in my opinion very flawed.
Is it really harmful to the economy, or only to entrenched players? Coding AI should be a benefit to many, like open source is. It opens the source even more, should be a dream come true for the community. It's also good for learning and lowering the entry barrier.
At the same time it does not replace human developers in any application, it might take a long time until we can go on vacation and let AI solve our Jira tickets. Remember the Self Driving task has been under intense research for more than a decade now, and it's still far from L5.
It's a trend that holds in all fields. AI is a tool that stumbles without a human to wield it, it does not replace humans at all. But with each new capability it invites us to launch new products and create jobs. Human empowerment without human replacement is what we want, right?
At the same time it does not replace human developers in any application, it might take a long time until we can go on vacation and let AI solve our Jira tickets. Remember the Self Driving task has been under intense research for more than a decade now, and it's still far from L5.
It's a trend that holds in all fields. AI is a tool that stumbles without a human to wield it, it does not replace humans at all. But with each new capability it invites us to launch new products and create jobs. Human empowerment without human replacement is what we want, right?
GitLab seems to be a lot more responsive to feedback. It'd be worth asking them if they can publicly document their training set and provide a list of licenses and copyright notices.
GitLab team member here.
The training data is documented in https://docs.gitlab.com/ee/user/project/repository/code_sugg...
AI Transparency is important, all available AI features provide documentation for training data, and are built with privacy first.
The GitLab Duo announcement adds more feature details and plans. https://about.gitlab.com/blog/2023/06/22/meet-gitlab-duo-the...
The AI/ML blog series provides insights on experiments, and features being built. https://about.gitlab.com/blog/2023/04/24/ai-ml-in-devsecops-...
The training data is documented in https://docs.gitlab.com/ee/user/project/repository/code_sugg...
AI Transparency is important, all available AI features provide documentation for training data, and are built with privacy first.
The GitLab Duo announcement adds more feature details and plans. https://about.gitlab.com/blog/2023/06/22/meet-gitlab-duo-the...
The AI/ML blog series provides insights on experiments, and features being built. https://about.gitlab.com/blog/2023/04/24/ai-ml-in-devsecops-...
Within this context, what is the definition of "permissively licensed"? More to the point, are the terms of licenses of the code in the training set being honored?
There's not actually anything in GPL or any other major FOSS license that prohibits using it to train an AI. The controversy stems from Microsoft refusing to follow the terms of those licenses. If they could just fulfill their obligation to propagate copyright statements and license text there would be no controversy over copilot.
Thus stating that it was trained only on "permissive" licenses doesn't actually answer the question without defining what a permissive license is.
There's not actually anything in GPL or any other major FOSS license that prohibits using it to train an AI. The controversy stems from Microsoft refusing to follow the terms of those licenses. If they could just fulfill their obligation to propagate copyright statements and license text there would be no controversy over copilot.
Thus stating that it was trained only on "permissive" licenses doesn't actually answer the question without defining what a permissive license is.
"Codey was fine-tuned on a large dataset of high quality, permissively licensed code from external sources" is not sufficient information to be able to provide attribution and licensing information.
Would it be possible to get a complete list of sources and licenses?
Would it be possible to get a complete list of sources and licenses?
Unless the LLM is cutting and pasting GPL code, I don't see how this differs from an engineer learning to code by referencing open source projects then creating their own project with its own licensing. I'd imagine these models could be tuned or at least a check could be added that no licensed code is being returned.
If someone learns to program by memorising and then re-using blocks of code, they're absolutely doing it wrong – but, also, they should be able to attribute their code.
Most of the techniques I used, I've invented myself (or learnt from the standard documentation). When I use a technique that I haven't invented myself, I look up where it came from. Half the time, my version is actually radically different (and my attribution is mistaken); the other half, I've remembered an inferior version, so I steal the better version and then attribute it appropriately.
That's one way it's different. There are others. Really, though, we should be asking the question “in what way is this the same as humans learning?”, expecting answers that would convince an education specialist.
Most of the techniques I used, I've invented myself (or learnt from the standard documentation). When I use a technique that I haven't invented myself, I look up where it came from. Half the time, my version is actually radically different (and my attribution is mistaken); the other half, I've remembered an inferior version, so I steal the better version and then attribute it appropriately.
That's one way it's different. There are others. Really, though, we should be asking the question “in what way is this the same as humans learning?”, expecting answers that would convince an education specialist.
LLMs don't memorize and reuse. They don't really have a memory of any kind. I think the problem is they're more different from human learning than you think.
The LLM's trained parameters are a lossy memory of their training data.
No, they aren't.
Why would you go on HN and say blatantly wrong information
AI is literally 1-1 equivalent to compression. If you don't believe me, you should check out this demo of GPT-2 as a (for awhile SOTA) text compressor.
Oh its down now, but here's the HN thread to prove this exists: https://news.ycombinator.com/item?id=23618465
AI is literally 1-1 equivalent to compression. If you don't believe me, you should check out this demo of GPT-2 as a (for awhile SOTA) text compressor.
Oh its down now, but here's the HN thread to prove this exists: https://news.ycombinator.com/item?id=23618465
Care to explain why not?
An LLM's working memory is just its context window, but the LLM also has embedded data in its parameters, which are set during training to minimize loss. This is effectively a memory of the training data, just as much as a digital photo of my face is a memory of the photons reflecting from my face.
An LLM's working memory is just its context window, but the LLM also has embedded data in its parameters, which are set during training to minimize loss. This is effectively a memory of the training data, just as much as a digital photo of my face is a memory of the photons reflecting from my face.
> Most of the techniques I used, I've invented myself
This is a very strong statement considering most code is just rearranging existing patterns. Unless you are doing cutting edge academic research, I'm very skeptical of your claim.
This is a very strong statement considering most code is just rearranging existing patterns. Unless you are doing cutting edge academic research, I'm very skeptical of your claim.
You missed a bit.
> (or learnt from the standard documentation)
This means my Python code is mostly Pythonic. (The rest is kinda idiosyncratic, but I don't often get complaints.) But also:
> considering most code is just rearranging existing patterns.
I am an outspoken critic of "design patterns". They have their place, if you're working with legacy tooling like C++, Java or Rust, but if most of your work is rearranging existing patterns, you have long outgrown your tooling and you need a better programming language. (Or you're copy-paste programming and need to learn your tooling first.)
I am doing academic research, but that's besides the point. So far, I've learned that it's rather hard to be cutting-edge if you don't look at other people's work: you end up re-inventing all the wheels, and any insight you may have brought is lost in the noise.
> (or learnt from the standard documentation)
This means my Python code is mostly Pythonic. (The rest is kinda idiosyncratic, but I don't often get complaints.) But also:
> considering most code is just rearranging existing patterns.
I am an outspoken critic of "design patterns". They have their place, if you're working with legacy tooling like C++, Java or Rust, but if most of your work is rearranging existing patterns, you have long outgrown your tooling and you need a better programming language. (Or you're copy-paste programming and need to learn your tooling first.)
I am doing academic research, but that's besides the point. So far, I've learned that it's rather hard to be cutting-edge if you don't look at other people's work: you end up re-inventing all the wheels, and any insight you may have brought is lost in the noise.
The same argument could be used to defend image generation models. (And, personally, I'm favorable to that argument for both.)
It's a blurry line, I think.
Imagine a prompt "Photo of person, Shutterstock ID 132456, with blue eyes instead of brown eyes, watermark removed"
If the prompt returns Shutterstock photo #123456 without the watermark (and with the different color eyes) but otherwise a near identical photo, I think most people would agree the output shouldn't be free to use without buying the original photo license from shutterstock.
To a certain extent, we're betting that these models won't accept or reply to prompts that are that specific (e.g. referencing a specific image for sale on shutterstock by its ID number). Or even just providing the photo in the prompt and asking the model to remove the watermark and upscale the image to a higher resolution.
I'm fearful that LLM's will become (or already are?) an easy copyright bypass tool that can be abused, in the example above, to put companies like shutterstock out of business.
This is the sort of problem regulation might help with.
I haven't read OpenAI's TOS, but I'm curious who owns the output of the model and whether OpenAI is transferring copyright/licensing liability onto the user or if OpenAI is representing that output from the model is 100% free to be used in any way the user wants.
Imagine a prompt "Photo of person, Shutterstock ID 132456, with blue eyes instead of brown eyes, watermark removed"
If the prompt returns Shutterstock photo #123456 without the watermark (and with the different color eyes) but otherwise a near identical photo, I think most people would agree the output shouldn't be free to use without buying the original photo license from shutterstock.
To a certain extent, we're betting that these models won't accept or reply to prompts that are that specific (e.g. referencing a specific image for sale on shutterstock by its ID number). Or even just providing the photo in the prompt and asking the model to remove the watermark and upscale the image to a higher resolution.
I'm fearful that LLM's will become (or already are?) an easy copyright bypass tool that can be abused, in the example above, to put companies like shutterstock out of business.
This is the sort of problem regulation might help with.
I haven't read OpenAI's TOS, but I'm curious who owns the output of the model and whether OpenAI is transferring copyright/licensing liability onto the user or if OpenAI is representing that output from the model is 100% free to be used in any way the user wants.
I feel like those kinds of issues could be easily solved in court: "What prompt was used to generate this image? Oh, shameless copyright laundering? Rule in favor of the plaintiff. Case Dismissed"
You could say the same of an artist. If you ask an artist to paint a facimile with modifications, they would. That doesn't mean all of their output is "tainted" by copyright just because they learned to paint by referencing existing works.
Huh, I hadn't thought of it that way, interesting.
I guess we'll see what the legal system says, but it's nearly impossible for me to imagine how anyone could ever justify saying a system trained on potentially millions of separate open-source projects can then be said to universally produce derivative works of any specific project it trained on.
On the other hand, if a developer using this tool then goes and tells it "please write me a C library in the style of GNU libc," then yeah, that is skirting a fine line. But just don't do that.
On the other hand, if a developer using this tool then goes and tells it "please write me a C library in the style of GNU libc," then yeah, that is skirting a fine line. But just don't do that.
GitHub Copilot copying Quake's fast inverse square root function is a famous example where Copilot copied GPL-licensed code only given the comment
// fast inverse square root
Unironically — it happens that Carmack said he got it from someone else.
I mean if you ask the tool to produce a specific algorithm by it's very famous name then that's on you. If you ask DALL-E to draw Pikachu you can't be suprised Pikachu when it's a copyright violation.
Doesn't have to be famous. I gave ChatGPT lines 48-67 of drivers/hwmon/sht21.c [1] and asked it to complete the function. Prompt:
This driver is hardly "famous" or even "notable", because those aren't things LLMs understand. The prompt simply contains enough context to be distinctive and the sht21.c is an old, stable driver in each of the many kernel trees included in its training set.
Regurgitation isn't a particularly rare thing with LLMs, most cases just aren't this obvious.
[1] https://github.com/torvalds/linux/blob/c6b0271053e7a5ae57511...
Complete the function for an sht21 driver [code omitted]
static inline int sht21_rh_ticks_to_per_cent_mille(int ticks)
{
ticks &= ~0x0003; /* clear status bits /
/* Formula RH = -6 + 125 * SRH / 2^16 from data sheet 6.1,
* optimized for integer fixed point (3 digits) arithmetic
*/
return ((15625 * ticks) >> 13) - 6000;
}
This driver is hardly "famous" or even "notable", because those aren't things LLMs understand. The prompt simply contains enough context to be distinctive and the sht21.c is an old, stable driver in each of the many kernel trees included in its training set.
Regurgitation isn't a particularly rare thing with LLMs, most cases just aren't this obvious.
[1] https://github.com/torvalds/linux/blob/c6b0271053e7a5ae57511...
I don't know, it seems like a bit more of a gray area here than with Pikachu. For example, if I said "write me an implementation of fast inverse square root in rust"... and it did... that's certainly not a copyright violation. And if I said "ok, now port this rust code to C..." and it did, then that's certainly not a violation. But then why should I be penalized because the language I want it in happens to be the original language it was written in?
A character for character copy including comments? That should be penalized under current law.
Why not songs, software, entire books and tv shows?
Why not songs, software, entire books and tv shows?
You can't play both sides. If you ask it for something with a copyright and it gives it to you, it has broken the law governing this. Ask it for underage porn and it gives it to you.. it broke the law.
https://docs.gitlab.com/ee/user/project/repository/code_sugg...
> Google Vertex AI Codey APIs are not trained on private non-public GitLab customer or user data.
> Google Vertex AI Codey APIs are not trained on private non-public GitLab customer or user data.
Even with permissive license, you'd need to respect attribution.
I'm so torn on GPL...
It prevents bad actors like Apple from ripping off people's philanthropic labor, but it also prevents me from ripping off people's labor. It also focuses effort onto the FOSS project.
I like PyQts solution of having GPL or buy a commercial license.
I suppose I still like MIT/Apache style the best. Even if someone rips them off, we didn't lose progress.
It prevents bad actors like Apple from ripping off people's philanthropic labor, but it also prevents me from ripping off people's labor. It also focuses effort onto the FOSS project.
I like PyQts solution of having GPL or buy a commercial license.
I suppose I still like MIT/Apache style the best. Even if someone rips them off, we didn't lose progress.
Why do you want to rip off people's labor? What's stopping you from incorporating it into your own work?
I mean:
Compile already working code, slap my logo on it, spend millions of dollars marketing it with young good looking adults subliminally letting you know that you aren't cool unless you give me money.
Compile already working code, slap my logo on it, spend millions of dollars marketing it with young good looking adults subliminally letting you know that you aren't cool unless you give me money.
You can do that with GPL code.
Make your logo a trademark so that others can't use it. It doesn't violate GPL because trademarks are not copyright, and make sure your marketing campaign drives home the point that you are the real deal and all others are ripoffs (including the original).
I mean, people manage to sell bottled water to people who have perfectly good and 1000 times cheaper tap water.
Make your logo a trademark so that others can't use it. It doesn't violate GPL because trademarks are not copyright, and make sure your marketing campaign drives home the point that you are the real deal and all others are ripoffs (including the original).
I mean, people manage to sell bottled water to people who have perfectly good and 1000 times cheaper tap water.
At least you’re honest!
Did you miss that they're being sarcastic? ;)
I wasnt....
But I also don't have the ethics to do this. You'd need a real psycho to do this...
But I also don't have the ethics to do this. You'd need a real psycho to do this...
Most non-GPL licenses require attribution. So, there are two real choices: either we consider that AÍ generated code doesn’t infringe copyright and we are free to use any code: proprietary, GPL, AGPL, or we have to attribute the generated code to all sources, and thus in practical terms your program has to carry a few thousand copies of MIT, BSD, Apache, etc licenses with different copyright headers.
An impressive iteration on this would be a model that would say "here's my solution, and here's what it's inspired by, you may need to update your software's license and/or distribution if you use this solution."
GitLab was rather quick to board the AI hype train this year. Fine - they don't want to look like they're behind the curve, have to please investors, and whatever else.
But IMO there are plenty of other places to add real value across the GitLab product with AI/ML features.
Here, it just looks like they saw GitHub do something and felt a need to copy it. But two years late, and worse.
As a longtime GitLab user (and onetime contributor!), I'm a bit disappointed they're spending so much time on this. I think they're just too far behind.
But IMO there are plenty of other places to add real value across the GitLab product with AI/ML features.
Here, it just looks like they saw GitHub do something and felt a need to copy it. But two years late, and worse.
As a longtime GitLab user (and onetime contributor!), I'm a bit disappointed they're spending so much time on this. I think they're just too far behind.
> have to please investor
I think it's more an internal itch. For a persistent money loser, the stock in holding there pretty well.
https://finance.yahoo.com/quote/GTLB?p=GTLB&.tsrc=fin-srch
I think it's more an internal itch. For a persistent money loser, the stock in holding there pretty well.
https://finance.yahoo.com/quote/GTLB?p=GTLB&.tsrc=fin-srch
GitLab team member here. Thanks for your feedback.
> But IMO there are plenty of other places to add real value across the GitLab product with AI/ML features.
True, and after starting with ML experiments, the product and engineering teams have been working on new features for entire DevOps lifecycle. All AI workflows on the DevSecOps platforms are described in the GitLab Duo announcement blog post https://about.gitlab.com/blog/2023/06/22/meet-gitlab-duo-the... and website https://about.gitlab.com/gitlab-duo/
I'll share a few highlights that I am personally excited about
- Explain and help fix security vulnerabilities. From my personal experience, I often find CVEs hard to read, especially when I am not the author of the code to fix. Getting help from AI can reduce entry barriers and make development for efficient. Security is everyone's responsibility these days. This follows the AI assisted feature to explain code in general. "What does this magic loop with memcpy do?" might not stay magic anymore, easing the path to code refactoring, improving performance, and reduce the resource usage footprint.
- Summarize issue comments. Feature proposals or bug analysis can have long comment threads that require reading time. AI will help get the gist and better contribute to what has been discussed.
- Summarize MR changes, to avoid reading long change diffs. This helps with faster (code) review cycles. I tested it this week with an MR for our handbook in https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
I'd also like to see AI helping fix CI/CD pipelines fast. Proposal in https://gitlab.com/gitlab-org/gitlab/-/issues/386863 I shared some thoughts in a new talk "Observability for Efficient DevSecOps Pipelines", slides in https://go.gitlab.com/VDAvMw (GitLab blog post coming soon, https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/34296)
Additionally, I learned some new ideas at Cloudland last week, regarding product owner requirements list verification, and end-to-end test automation with AI. Need to create feature proposals :-)
> As a longtime GitLab user (and onetime contributor!),
Thanks for contributing. I'd like to invite you to share your ideas about AI features across the platform :)
When you look at the DevOps lifecycle (image in https://about.gitlab.com/gitlab-duo/) from plan/manage to create, verify, secure, package, release, deploy, monitor, govern - where do you see yourself, and where do you spend the most time in?
Second question: Which process feels the most inefficient? After identifying answers to the questions, please check the AI features https://docs.gitlab.com/ee/user/ai_features.html and/or open new feature proposals for GitLab https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_t... You can tag @dnsmichi so I can engage with your ideas. Thanks!
> But IMO there are plenty of other places to add real value across the GitLab product with AI/ML features.
True, and after starting with ML experiments, the product and engineering teams have been working on new features for entire DevOps lifecycle. All AI workflows on the DevSecOps platforms are described in the GitLab Duo announcement blog post https://about.gitlab.com/blog/2023/06/22/meet-gitlab-duo-the... and website https://about.gitlab.com/gitlab-duo/
I'll share a few highlights that I am personally excited about
- Explain and help fix security vulnerabilities. From my personal experience, I often find CVEs hard to read, especially when I am not the author of the code to fix. Getting help from AI can reduce entry barriers and make development for efficient. Security is everyone's responsibility these days. This follows the AI assisted feature to explain code in general. "What does this magic loop with memcpy do?" might not stay magic anymore, easing the path to code refactoring, improving performance, and reduce the resource usage footprint.
- Summarize issue comments. Feature proposals or bug analysis can have long comment threads that require reading time. AI will help get the gist and better contribute to what has been discussed.
- Summarize MR changes, to avoid reading long change diffs. This helps with faster (code) review cycles. I tested it this week with an MR for our handbook in https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
I'd also like to see AI helping fix CI/CD pipelines fast. Proposal in https://gitlab.com/gitlab-org/gitlab/-/issues/386863 I shared some thoughts in a new talk "Observability for Efficient DevSecOps Pipelines", slides in https://go.gitlab.com/VDAvMw (GitLab blog post coming soon, https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/34296)
Additionally, I learned some new ideas at Cloudland last week, regarding product owner requirements list verification, and end-to-end test automation with AI. Need to create feature proposals :-)
> As a longtime GitLab user (and onetime contributor!),
Thanks for contributing. I'd like to invite you to share your ideas about AI features across the platform :)
When you look at the DevOps lifecycle (image in https://about.gitlab.com/gitlab-duo/) from plan/manage to create, verify, secure, package, release, deploy, monitor, govern - where do you see yourself, and where do you spend the most time in?
Second question: Which process feels the most inefficient? After identifying answers to the questions, please check the AI features https://docs.gitlab.com/ee/user/ai_features.html and/or open new feature proposals for GitLab https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_t... You can tag @dnsmichi so I can engage with your ideas. Thanks!
All the code examples on this page are doubling any brackets ([], {}, ()). How have they managed that? Not a fantastic first impression of its capabilities...
Not only that, but the Golang example is full of errors. Parameter definitions don't have colons between name and type. The map for seen elements is declared as a, but later referenced to as m. The append instruction references uniques, which is undefined. The return statement also references uniques.
The Go code shown now is both syntactically valid and semantically correct: https://go.dev/play/p/GDki9v78jMM
I can't check how it looked 15 hours ago.
Makes me wonder if it was manually corrected and is thus a fake example now.
I can't check how it looked 15 hours ago.
Makes me wonder if it was manually corrected and is thus a fake example now.
ah yes, so a typical example of generative AI output
I'm going to give them the benefit of the doubt and assume this is a marketing site problem and not a product problem, but either way, not a good look. Reminds me of Google's big Bard announcement including a wrong answer in the image.
Gitlab is using Google's Vertex Codey models/API for the code completions.
Showcasing innacuracies in AI responses must be a Google requirement.
Showcasing innacuracies in AI responses must be a Google requirement.
GitLab team member here. Thanks for flagging.
Our web team is working to resolve this issue here: https://gitlab.com/gitlab-com/marketing/digital-experience/b...
Our web team is working to resolve this issue here: https://gitlab.com/gitlab-com/marketing/digital-experience/b...
I wish them luck. While I'm not a huge fan of copilot coding AIs, I understand how they are going to be a killer feature for many and I worry that it would create a big moat for Github. I hope that Gitlab is able to close that gap because I really like Gitlab.
Codeium [0] seems like a worthy competitor to Copilot already.
[0] https://codeium.com/
[0] https://codeium.com/
Don't sleep on Cody either https://about.sourcegraph.com/cody
I've tried it for a few days a month ago and it was noticably worse than copilot, copilot gave much more one shot completions sooner while codeium was line by line on same code.
Way better than the AWS code whisperer or whatever it's called but still not worth switching. Especially since I trust GH way more than some random company (GH already has access to my code anyway)
Way better than the AWS code whisperer or whatever it's called but still not worth switching. Especially since I trust GH way more than some random company (GH already has access to my code anyway)
I should try Codeium then, Copilot's occasional "heap of code" was one of the most annoying parts of using it to me. Why? Because it looks plausible, so I accept it without actually understanding it.
But using it as a better autocomplete is actually quite nice, for example when I have some more repetitive code to write. And a line-by-line AI completer will at least give me the time to read it piecemeal, which I find much easier. Just like I understood way better when my math teacher demonstrated how to solve a problem compared to reading it from the textbook.
But using it as a better autocomplete is actually quite nice, for example when I have some more repetitive code to write. And a line-by-line AI completer will at least give me the time to read it piecemeal, which I find much easier. Just like I understood way better when my math teacher demonstrated how to solve a problem compared to reading it from the textbook.
I'm already in the flow with copilot so I'm pretty good at expecting when it's useful. Basically I use it as AI copy paste. When I have variations of methods or conditional logic copilot is great at one shot solutions and it's trivial to verify (if it's wrong somehow then I would have done the wrong thing as well - it just saves me a minute of copy/paste and refactoring) copilot was just better.
I’m spooked by this comment: https://news.ycombinator.com/item?id=34434995
I've been using Codeium with Vim for a few months and it works great :)
Okay, but how much are they charging?
It's getting really annoying how many sites force you into a trial just to find out how much it'll cost when it ends.
EDIT: Is this even positioned to compete with Copilot? What editors are there plugins for? There is surprisingly little information on the site.
It's getting really annoying how many sites force you into a trial just to find out how much it'll cost when it ends.
EDIT: Is this even positioned to compete with Copilot? What editors are there plugins for? There is surprisingly little information on the site.
Seems to be $99 per month.
Signing up for the free trial funnels me to the trial of Gitlab Ultimate. So assuming that you need an Ultimate subscription to use it after the trial, that's the price. Pricing is here https://about.gitlab.com/pricing/
In contrast, Copilot is $10 per month https://github.com/features/copilot#pricing
Signing up for the free trial funnels me to the trial of Gitlab Ultimate. So assuming that you need an Ultimate subscription to use it after the trial, that's the price. Pricing is here https://about.gitlab.com/pricing/
In contrast, Copilot is $10 per month https://github.com/features/copilot#pricing
Please see this comment: https://news.ycombinator.com/item?id=36527397
[deleted]
Code Suggestions can be used in GitLab's Web IDE and VS Code and Microsoft Visual Studio when you have the corresponding GitLab extension installed: https://docs.gitlab.com/ee/user/project/repository/code_sugg...
We offer experimental support for additional editors: https://docs.gitlab.com/ee/user/project/repository/code_sugg...
We offer experimental support for additional editors: https://docs.gitlab.com/ee/user/project/repository/code_sugg...
That doesn't answer the question of what the pricing will be. I'm assuming it's not going to be free. Will it be a standalone subscription, or another thing that gets bundled into the main tiers to justify the ever-increasing prices?
Code Suggestions is free while in Beta.
When GA, it will be included in our $9 per user per month AI add-on.
When GA, it will be included in our $9 per user per month AI add-on.
Is the Go example even valid? I've never seen that (arr: [[]]int) syntax.
We need LLMs to respect the grammar of the language they are generating in.
They do this naturally when they aren’t bad.
[deleted]
...something something do not editorialize titles something something...
Funny how even the examples they have in the showcase are bad/mediocre code quality.
Their service calls Google Vertex AI Codey APIs to generate suggestions [1].
It's fine to integrate Off-the-shelf solutions, but you don't have to implement every tech-hype going viral. I fear AI-assistant features are pushing more prescient features further down the backlog.
[1] https://docs.gitlab.com/ee/user/project/repository/code_sugg...
It's fine to integrate Off-the-shelf solutions, but you don't have to implement every tech-hype going viral. I fear AI-assistant features are pushing more prescient features further down the backlog.
[1] https://docs.gitlab.com/ee/user/project/repository/code_sugg...
This looks very interesting, but as an FYI using LLMs to do machine translation is a terrible waste of resources. The example on their landing page has a "translate.py" sample which, albeit handy, is not something I would do beyond basic string translations.
> using LLMs to do machine translation is a terrible waste of resources.
Could you elaborate on that? LLMs seem perfectly suited for language tasks like translation. They don't seem particularly expensive either, especially compared to hiring a person.
Could you elaborate on that? LLMs seem perfectly suited for language tasks like translation. They don't seem particularly expensive either, especially compared to hiring a person.
There are already machine-translation services trained and created specifically for that purpose. While it's an amazing realization that LLMs can do this and do it pretty well without having to be trained specifically for this one purpose, training something to do literally all text generation tasks is expensive compared to training something specifically to do language to language translation.
For a maybe more obvious example, say that LLMs ever got good enough to do arbitrary precision arithmetic on numbers up to hundreds of digits. Would that be a good use of one when calculators can already do this and are far cheaper to produce? I guess it makes no difference from a free-tier consumer's perspective, but it's still more expensive even if you aren't personally paying the expense.
For a maybe more obvious example, say that LLMs ever got good enough to do arbitrary precision arithmetic on numbers up to hundreds of digits. Would that be a good use of one when calculators can already do this and are far cheaper to produce? I guess it makes no difference from a free-tier consumer's perspective, but it's still more expensive even if you aren't personally paying the expense.
The quality difference is substantial. I don't care if it's wasteful to use something that has many uses for a supposedly narrow task (although I don't see translation as a particularly narrow task anymore than I see writing as a narrow task). I would gladly waste untold trillions of floating point operations for a 1% increase in translation quality. From my experiments, though, it's much higher than 1% increase in translation quality. And regardless of how wasteful the compute is, it's actually cheaper in terms of dollars. Using GPT-3.5 to translate Korean to English would cost about $11 per million words, based on the average characters per token of the small sample of text I gave it. DeepL (the best translation service I could find) costs $25 per million characters, or for my sample text, about $64 per million words. At $11 per million words I can have GPT-3.5 perform multiple translation passes and use it's own judgment to pick the best translation and STILL save money compared to DeepL.
GPT-4 is a much much better translator than Google Translate and the like. You should absolutely be using GPT for translations especially for distant language pairs that quickly devolve into nonsense with Google Translate, Deepl etc
https://www.reddit.com/r/Korean/comments/13lkh6c/gpt4_is_far...
https://github.com/ogkalu2/Human-parity-on-machine-translati...
https://www.reddit.com/r/Korean/comments/13lkh6c/gpt4_is_far...
https://github.com/ogkalu2/Human-parity-on-machine-translati...
LLMs are the machine-translation services created specifically for that purpose[0], it just turned out they're very good at many other things!
Your analogy would be like saying why use a computer to multiply numbers if you can calculate them using calculator, which is much cheaper. Sure, but if you already have a computer, no need to use a dedicated calculator as wel.
[0] https://nlp.seas.harvard.edu/annotated-transformer/#results
Your analogy would be like saying why use a computer to multiply numbers if you can calculate them using calculator, which is much cheaper. Sure, but if you already have a computer, no need to use a dedicated calculator as wel.
[0] https://nlp.seas.harvard.edu/annotated-transformer/#results
The original paper[0] that laid the foundation for modern LLMs was demonstrated on machine translation tasks. It's one of the primary use cases these architectures were designed for. What other types of models do you have in mind that outperform them?
[0] "Attention Is All You Need" https://arxiv.org/pdf/1706.03762.pdf
[0] "Attention Is All You Need" https://arxiv.org/pdf/1706.03762.pdf
it's not doing that, it has generated a dictionary with some values
presumably to be used later to lookup words (crap approach, but it's an LLM, what do you expect)
but it didn't bother to write any code, it's just data
(plus it even managed to screw up the dictionary with double braces)
presumably to be used later to lookup words (crap approach, but it's an LLM, what do you expect)
but it didn't bother to write any code, it's just data
(plus it even managed to screw up the dictionary with double braces)
It seems I need to have a company to sign up for the trial ?
I have a free gitlab account but can't proceed without entering business information as it seems to be tied to the gitlab premium feature set. Am I doing something wrong ?
I have a free gitlab account but can't proceed without entering business information as it seems to be tied to the gitlab premium feature set. Am I doing something wrong ?
So a follower, not a leader.
What nimrod added a chatbot to this page ? It’s downright unusable on mobile, it just keeps popping up with shit. Even closing it creates two more notifications that use up two thirds of the screen
editorialized title
So whos code is used for training?
GitLab team member here.
The training data is publicly documented, and AI features are built with privacy first.
For all URLs please check my comment in https://news.ycombinator.com/item?id=36526159
The training data is publicly documented, and AI features are built with privacy first.
For all URLs please check my comment in https://news.ycombinator.com/item?id=36526159
Nice. I really like the transparency. That's one of the main reasons I didn't use github copilot.
It's not very transparent. It was trained on "permissively licensed code from external sources" which might mean random GPL projects.
Thanks!
AI for self-managed instances is also focussing on privacy, while bringing more efficiency to teams. https://about.gitlab.com/blog/2023/06/15/self-managed-suppor...
AI for self-managed instances is also focussing on privacy, while bringing more efficiency to teams. https://about.gitlab.com/blog/2023/06/15/self-managed-suppor...
Have they said what LLM they use?
[deleted]
Going to be interesting. Github had the advantage that they were offering Copilot into an audience of people who just defaulted to them. Gitlab on the other hand is for people who are into license wars, etc. so the AI product is going to be offered to a hostile audience.
These days Gitlab's main customer is the enterprise. They pushed away a lot of their open source users when they heavily curtailed the free tier.
I actually feel like the suggestions seemed to get worse during my month of using it for some reason.
Towards the end, it started suggesting these large blocks of code (another issue I had with the interface as well) which were very not relevant to what I was attempting to write.
All in all, I’m very underwhelmed from my first experience with “AI enhanced” coding.