my understanding is that yes it is, you cant make shoes in the US, but the power that comes from pretty much all finance flowing through the American pipes is a good trade off.
I wonder how this compares to 'catastrophic forgetting' that can be a problem of full fine tuning. Or at least that's what I've just been reading as a case _for_ using LoRa, as it's not susceptible to that. I guess this paper shows LoRa causes forgetting in a different way.
Are there good general principles yet for what fine tuning method to use in certain situations? It still seems quite difficult to know ahead of time what's going to happen.
I wonder, is this exponential relation specific to multi-modal models? From my admittedly naïve view it seems to make sense that "...what is rare is not properly learned" would apply generally?
Towards the end they state: ‘… just adding “do not hallucinate” has been shown to reduce the odds a model hallucinates.’ I find this surprising and doesn’t fit with my understanding of how a language model works. But I’m very much a novice. Would this be due to update training including feedback that marks bad responses with the term “hallucinate”?
Undoubtedly that can be the cause in some cases, but there are counterexamples. Like West Virginia which has a bad opioid addiction problem yet relatively low homeless rate. What does appear highly correlated is homeless rates vs. cost of housing to income ratios. Find a city with a real estate bubble, and you'll likely find a large tent city too.