I’ve had opiates for wisdom tooth extraction and a couple of relatively minor but very painful injuries. I was very glad to have them. They made the experiences much more bearable. I did not experience any addiction issues, and I’m glad that my access to these medications was not limited due to some people’s inability to handle them.
I really don’t think forcing needless suffering on everyone is the answer to this problem.
And, with that, we’re back to pre-LLM chatbot design: intent classification, entity extraction, business logic, return a result. Only the whole process rests on a more rickety foundation. It’s also bloated and slow, querying an LLM over and over for these things. I’m starting to see some parallels to modern JavaScript and SPAs. ;-)
Bard hasn’t been using Google’s best language models. I believe it just got an upgrade, however, and I’m now getting output that is significantly more coherent and useful than ChatGPT’s. It’s also a helluva lot faster, though that could owe to the limited access.
That’s only going to get you so far. Sparsely represented subjects have have an actual reality you’re trying to model. So while yes, you can generate synthetic data to interpolate between the points that have already been sampled, whether those interpolated points have anything to do with the underlying reality is a matter of chance. If you really try to drill down the existing tools into a specialized problem area, it becomes very clear that they lack a sufficiently informed model of the subject to be useful. That’s why I’ve found that ChatGPT is great at the beginning of a project and increasingly irrelevant as you approach the core technical challenges of the project.
OpenAI’s fear-mongering efforts have been really transparent. As an example, in the ABC News piece, one of their employees discusses asking GPT to help build a bomb. Your employees using words like “bomb” on television is not something that happens by accident.
OpenAI explicitly disclaims ownership interest in the model outputs. A user who both generates outputs from OpenAI AND uses it to train a “foundational” model that competes with OpenAI could owe contract damages. Other parties? I simply don’t see it.
I think a review of the state of frontend tooling will show that efficiency with respect to developer hours is not a widely shared priority. I only say this with only 50% intention of starting a flame war.
Any news on whether they're pursuing downstream weights? That's a really interesting question for me - if you started at the LLaMA weights and arrived somewhere else by supplying additional training data and computation, any infringement claim would be very, very murky. I also think it would be practically difficult to prosecute such a claim because it would necessarily involve comparative questions about the consumption of unlicensed training data to generate the original weights. It would certainly make for interesting law.
It’s not a lazy refrain, it’s a serious downside of the technology. It’s tiring and stressful to supervise the work product of an assistant that is extremely capable sometimes, but a compulsive bullshitter at other times. Just as it’s tiring to supervise an “autonomous” car that often deftly navigates the road, but sometimes wants to plow into a school bus.
A concern I have about OpenAI is that, if you're using their APIs to develop an application, they can mine your data to compete with you, or even beat you to market. They can do this indirectly, by sharing information with preferred business partners. The conflict of interest, combined with the lack of robust data privacy guarantees, makes me queasy.
If serving up generic LLM APIs becomes commoditized -- and I think it will -- they will want to monetize in other ways.
OpenAI disclaims ownership interest in the model output. If a subscriber (who has a contractual relationship with OpenAI) chooses to generate outputs that could be used to train a competing model, and chooses to share those outputs with third parties, that is not prohibited by the agreement. Further, the data being shared belongs to the subscriber and can be licensed however they desire (though actually, model outputs may not be copyrightable at all). If a third party who does not have a contractual relationship with OpenAI chooses to take this data and train a competing model, they are using the data under valid license and have breached no obligation to OpenAI.
A language model takes in a sequence of tokens and outputs a probability (0-1) for each token in the vocabulary (the set of all tokens the model knows). Based on this probability distribution, there are various sampling strategies that can be employed to choose which token to actually show to the user.
If we continue on this trajectory, I have a suspicion that the big players will increasingly cry “danger!” and, as Sam Altman has done already, call for government regulation of AI. Having potential upstarts buried in red tape is how monopolies and oligopolies sustain their positions in a lot of industries.
Hm. I haven’t tried the local installs yet. However, when the Alpaca web demo was live, I did find it to be comparable (though not quite as capable) to davinci-003. It answered arbitrary factual questions about pop culture references, law, medicine, and programming. It generated rhymes and poems. (I didn’t try asking for the emoji thing, so can’t say anything about that.) It performed natural language tasks such as information extraction and summarization. And it did all of it coherently.
I don’t put much stock in the claims about GPT4 “passing” professional exams. Many copies of previously administered exams are available, and the exams are formulaic in their construction (to make them stable, predictable targets).
A compressive copy of the internet brute-forcing its way through an exam (which it may even have digested already) is really not interpretable as performing well on the exam. It’s a meaningless measure because the tests were not designed with this use in mind.
Well, consider the source. These are the same people who withheld measly little GPT2 because it was too dangerous for the grubby masses. They profess to have all of this concern for the “safety” of their new technology, yet are simultaneously pouring rocket fuel on top of a wildfire of hype.
“For our initial run, fine-tuning a 7B LLaMA model took 3 hours on 8 80GB A100s, which costs less than $100 on most cloud compute providers. We note that training efficiency can be improved to further reduce the cost.”
($500 was what they paid OpenAI to generate the fine-tuning dataset.)
I really don’t think forcing needless suffering on everyone is the answer to this problem.