However, if the advantage is due to things like inference infrastructure to support a massive model, that isn't easy to duplicate.
I would also say that the quality of these smaller models are good, but we also may not be measuring them correctly. Recent papers suggest that these smaller LMs dont fully capture ChatGPT quality in ways that may not have appeared with crowd worker ratings [1]. It's easy to have your inputs be inside a happy distribution for a paper but fail in the real world in ways that GPT-4 doesnt.
Lmsys would love to compare with bigger models but have limited resources. Contributions are welcome [2]
We've got to stop calling this a "Google Memo." That's a false narrative. It's just a random doc written by one of 140000+ employees.
> Google has been contacted for comment but it is understood that the document is not an official company memo. [1]
There are moats to products, but less so to pure language models trained on the same web-scale scraped data that many share.
Not all data is readily available to language models, and integration can be difficult.
A company that specializes in say AI for trash sorting likely still has a moat.
Microsoft integrating AI into Windows still has a moat (for Windows).
GPT-4 is ~200 Elo better than the next best semi-public Vicuna-13B in Chatbot Arena [2]. That is a non-zero moat - perhaps due to hosting larger models, training data, licensing, output postprocessing, etc.
However, if the advantage is due to things like inference infrastructure to support a massive model, that isn't easy to duplicate.
I would also say that the quality of these smaller models are good, but we also may not be measuring them correctly. Recent papers suggest that these smaller LMs dont fully capture ChatGPT quality in ways that may not have appeared with crowd worker ratings [1]. It's easy to have your inputs be inside a happy distribution for a paper but fail in the real world in ways that GPT-4 doesnt.
Lmsys would love to compare with bigger models but have limited resources. Contributions are welcome [2]
[1] https://arxiv.org/abs/2305.15717
[2] https://lmsys.org/blog/2023-05-25-leaderboard/#next-steps