The 2B model seems underwhelming. For instance, compared to the recent StableLM2 1.6B model that is slightly smaller and probably wastes some "English metric points" by being multilingual.
The latter (and other similar open models) seem to do similarly well in benchmarks (much better in Math?) with way less fancy stuff. For instance, public data and no secretive filtering with pre trained models or synthetic data.
My take is that using the vanilla approaches take you really far. And many of the latest tricks and hours-of-work buy you little... Will be interesting to see how this plays out, especially for the open source community.
The latter (and other similar open models) seem to do similarly well in benchmarks (much better in Math?) with way less fancy stuff. For instance, public data and no secretive filtering with pre trained models or synthetic data.
My take is that using the vanilla approaches take you really far. And many of the latest tricks and hours-of-work buy you little... Will be interesting to see how this plays out, especially for the open source community.