Before post-ChatGPT boom, we used to talk of "catastrophic forgetting"...
Make sure the new training dataset is "large" by augmenting it with general data (see it as a sample of the original dataset), use PEFT techniques (freezing weights => less risks), use regularization (elastic weight consolidation).
Fine-tuning is fine, but will be more expensive that you thought and should be led by more experienced ML engineers. You probably don't need to fine tune models anyway.
The only valid argument I see is that some brutalist buildings are historically important and even we don't like them today we should keep some of them for future generations to visit and see what we have tried, and disliked.
But to use the environment as an excuse is silly. There are always things that can be done for that: recycling the concrete, rebuild to reduce car dependency and improve energy efficiency, choose more sustainaible materials (like wood) that can easily be replaced in the future (instead of more concrete), etc
ML has been around for decades, DL for more than a decade.
In 2019, I had to explain to executives that 95% of AI projects fail (based on some other survey), top 1 reason is bad or missing data and top 2 is misaligned internal processes. I probably still have the slides somewhere.
One project I worked on was impossible because the data was so bad that after cleaning, we went from 4M rows to 10k usable rows in 4 languages. We could have salvaged a lot more if we restricted the use case but then the benefits of the projects would be not so interesting anymore. The internal sponsor gave up and understood the problem. Instead, they decided to train everyone on how to improve data entry and quality! In just 6 months I could see the data was getting better indeed.
But I had to leave this company, the IT dep was too toxic.
So I think the author is right. According to Scale, we'd have gone from 95% failures to 95% successes in just 4-5 years just thanks to LLMs? This is of course ridiculous, knowing the problem was never poor models.
Started a career in ML/AI years before ChatGPT changed everything.
At the time, we only used the term AI if we referred more than just machine/deep learning techniques to create models or research something (thinks operations research, Monte Carlo simulations, etc).
But it started to change already.
I think startups and others will realise to make a product successful, you will need clean data and data engineers, the rest will fill follow. Fundamentals first.
All the startups trying to sell "AI" to traditional industries: good luck!
I've worked as an AI engineer for a big insurance, contractor with a bank, and oh gosh!
Make sure the new training dataset is "large" by augmenting it with general data (see it as a sample of the original dataset), use PEFT techniques (freezing weights => less risks), use regularization (elastic weight consolidation).
Fine-tuning is fine, but will be more expensive that you thought and should be led by more experienced ML engineers. You probably don't need to fine tune models anyway.