We're Gonna Need a Bigger Moat(steve-yegge.medium.com)
steve-yegge.medium.com
We're Gonna Need a Bigger Moat
https://steve-yegge.medium.com/were-gonna-need-a-bigger-moat-478a8df6a0d2
16 comments
I've always thought Steve Yegge wrote great essays, but this isn't compelling at all.
And sadly it just ends up being a sales pitch for his latest project.
And sadly it just ends up being a sales pitch for his latest project.
Not the first such post in his archives.
I'm not well versed in AI, as a disclaimer.
That said, as a user, the moat between ChatGPT and say, Bard seems as wide as the Pacific.
Put another way, in my tests giving both coding questions, I'd probably hire ChatGPT as a junior or intermediate developer. I'd probably tell Bard to choose a different profession, if not recommend classes for the mentally challenged.
If all of this were so fast moving and cheap, why is Bard so far behind?
That said, as a user, the moat between ChatGPT and say, Bard seems as wide as the Pacific.
Put another way, in my tests giving both coding questions, I'd probably hire ChatGPT as a junior or intermediate developer. I'd probably tell Bard to choose a different profession, if not recommend classes for the mentally challenged.
If all of this were so fast moving and cheap, why is Bard so far behind?
It is hard for me to compare two black boxes. I agree that ChatGPT looks leaps and bounds better than Bard, but it’s hard for me to tell the power of the core transformer model ends and the heuristics and secondary models begin.
Knowing what those little components look like, for me, would help me evaluate which model is actually better. Right now it feels to me that the difference between the two comes down to more copyrighted materials in ChatGPT’s training set… and more sophisticated ways to hide that, but that’s just my hunch.
“That fundamental formula has not really changed much for years,” says Jakub Pachocki, one of GPT-4’s developers. “But it’s still like building a spaceship, where you need to get all these little components right and make sure none of it breaks.”
https://www.technologyreview.com/2023/03/14/1069823/gpt-4-is...
Knowing what those little components look like, for me, would help me evaluate which model is actually better. Right now it feels to me that the difference between the two comes down to more copyrighted materials in ChatGPT’s training set… and more sophisticated ways to hide that, but that’s just my hunch.
“That fundamental formula has not really changed much for years,” says Jakub Pachocki, one of GPT-4’s developers. “But it’s still like building a spaceship, where you need to get all these little components right and make sure none of it breaks.”
https://www.technologyreview.com/2023/03/14/1069823/gpt-4-is...
Theory, ClosedAI (M$) has focused on coding to win over tech people, tech people tell average people to use their products and it snowballs.
Do you think it’s a coincidence Microsoft bought GitHub and all of a sudden they have a coding focused bot ? Weird right ? Think of all that sweet training material.
Do you think it’s a coincidence Microsoft bought GitHub and all of a sudden they have a coding focused bot ? Weird right ? Think of all that sweet training material.
In an ever-evolving AI landscape, data moats have become increasingly important. Companies that possess large amounts of generated human content, such as Microsoft's acquisition of GitHub, are likely to see an increase in value as a result of this development. The unique data generated by these platforms can give companies an advantage in training and fine-tuning AI models, potentially allowing them to stay ahead of the competition. In this context, a data moat serves as a protective barrier against competitors, ensuring that the company continues to hold a unique and valuable asset in the industry.
It's a fair theory. I think coding is so pivotal, or scary, because it starts to prove these things can write or fix themselves. I'm still not sure whether it's impressive, depressing, or scary.
Just answering questions is a glorified Ask Jeeves.
Just answering questions is a glorified Ask Jeeves.
Hmm not sure if it means it can self-improve itself exactly. It feels like a lot of variables involved there and often the code produced is just an approximation of an existing requirement / something in the training data.
But yeah, I’m not quite sure what the end goal of l this research is supposed to be either. Seems like we’re not acting very intelligently on many fronts.
But yeah, I’m not quite sure what the end goal of l this research is supposed to be either. Seems like we’re not acting very intelligently on many fronts.
>That said, as a user, the moat between ChatGPT and say, Bard seems as wide as the Pacific.
From my observation bard is better at sql than chatgpt
From my observation bard is better at sql than chatgpt
Bard even hallucinated when I asked about Youtube Data API v3.
If you can't properly answer about your own API's what good are you ?
If you can't properly answer about your own API's what good are you ?
This article seems out of touch with reality by a large degree. From what I am seeing on the ground people are bored with language models and moving on. Maybe the research is exciting to you but the results are at an obvious plateau and has been like this for more than a month.
A month is a very short time scale, but I agree that the next big step is probably not around the corner, and maybe a different paradigm altogether
Can you expand on this? A reference to an article or an interview or simply experience. I'm trying to wade through the AI hype storm (or reality), and this POV would be helpful.
> Inventions as significant as the Transformer are more discovery than invention…
Like the “invention” of fire?
Like the “invention” of fire?
It did say transformers are mathematical. A new set of axioms can invent a branch of math, then anything found to come from them would be discoveries.
Ok, so. Where to start?
Hmm… ok. Well, how abort we ignore all the rest of the stuff and cut to heart of this article:
Assertion 1) Vicuna 13B is as good as bard and chat gpt.
Assertion 2) LoRA can stack infinitely and every LoRA you stack on top is additive only, increasing capability knowledge with no downside.
I think you can make your own mind up if you choose to believe these assertions, and judge for yourself if the author is on the right track or not.
I will link however, to the actual comments by the authors of vicuna, specifically the limitations and evaluation section: https://lmsys.org/blog/2023-03-30-vicuna/#how-to-evaluate-a-...
Maybe read it yourself and make an informed opinion…
(Unlike, it appears, the author of this article)