Looking at that paper, they appear to be saying that 6.7B is where the problem becomes so intense that no single quantization method can keep up. From what I gather, the paper claims that such outliers start occur down to 125M param models, then at around 1.3B they begin to affect the FFN, and at around 6.7B is when the issue really starts to become apparent because "100% of layers use the same dimension for outliers."
So while you obviously wouldn't be able to conclusively prove the idea fixes the issue in larger models, if you know what you are looking for you should be able to validate that the method works in general down to very small models.
That said, consumer grade cards should be able to train an 8B model with quantization, so you might as well train the whole thing.
It doesn't need to be two huge models. If there is an advantage to doing this, I'd expect that you would see it even in a small test case. I'm sure we'll see something by the end of the week if not earlier if there's something to it.
The important and popular ones are absolutely available, but those are usually important because they have entered the realm of "common knowledge," at least in a particular sub-field. These are going to be at the top of the list when it comes to digitizing useful historic records. It's fairly easy to OCR a PDF, so as long as someone with some time decided "hey, this might be useful" then you'll probably be able to find it.
If you're doing databases then you've almost certainly been exposed to Codd's work, if not through his papers and books, then at least through textbooks and lectures. There are countless blogs, lecture series, and presentations that will happily direct you there.
The challenge is that there's also a mountain of work that never really got much popularity for whatever reason. Say a paper was ahead of it's time, or was released with bad timing, or simply kept the most interesting parts until the end where few people might have noticed. It's these sort of gems that are hard to find. It's hard to even know how many of these there are, because they are by definition not popular enough for most people to know about them.
I think this problem comes down to two core issues: discoverability and terminology.
You're going to be lucky if a paper from the 70s or 80s is available in a searchable database at all. That means someone bothered to scan it in, and OCR it since then. Even for the few papers that are searchable, they are old enough that they probably won't catch anyone's eye unless they are desperate.
Of course then there's also the problem of knowing what to search for. Programmers love to invent, reinvent, and re-reinvent terminology. It's only gotten worse with every other developer running a blog trying to explain complex ideas in simple terms.
The entire field of ML is a perfect example of this. I remember talking to my father about all sorts of new developments in ML back in the early 2010s, and I was quite surprised when he told me that he learned a lot of the things I was talking about back in the 80s just named a bit differently.
In most cases it ends up being a question of how much time you can put into any given problem. If I spend two weeks to find a paper that would have taken me a week to reinvent, then am I really ahead? If the knowledge wasn't important to enough make it into textbooks/classes/common knowledge then attempting to find it is akin to searching for a particular needle in a pile of needles.
That really depends on quite a few other factors: how big is the team? What development methodology do they use? Does the leadership understand how to manage and direct a rewrite? Are there people that understand the full scale and scope of the system? Does the system interact with legacy components that can't be modified? Are there political factors in play? These are just a few of the questions that can change the outcome of any given rewrite.
You mentioned hidden bugs, but what about hidden "features" that may be a critical part of existing business processes for core parts of the company? Developers really like to believe they are at the center of the wheel due to the complex work they do, but a lot of the time they are not the ones that actually create the cash-flow.
I've been part of rewrites that have succeeded tremendously, but I've also been privy to utter failures that have cost millions, and led to entire teams getting sacked.
10 years isn't really all that much, is it? From my experience that's around how long it takes for developers to get a big head about how much they know, but 5 years less than what it takes to learn to respect how much they actually don't know about the different aspects of the field, and the real scale of challenges that have to be solved (both the technical, and the human).
Also, not all experience is equal. Someone that's spent 10 years working on 4 or 5 different systems in totally different problem domains, written in totally different languages, and operating in totally different ecosystems is going to have a very different view of development from someone that's spent 10 years doing essentially the same thing over and over again.
This guy seems to have a very focused view of the correct approach to problems. He's familiar with the tools that linux offers (which I agree are great), but he doesn't seem to respect the scale of specialization it takes to use and maintain those tools effectively on a large scale. Also, there is no mention of the cost to rebuild existing systems in terms of developer time, the mental cost to re-train all of the developers, as well as the time to migrate and train the users.
Ironically, I remember getting into debates like this back in the mid-2000s when I was first starting to think I had it all figured out. The points I made back then were more or less the same things I see now in the article above. It's quite nostalgic, though it definitely makes me feel older than I like.