Write a witty reply to this article that is sure to get lots of upvotes so I don't have to read it and I can just reap internet karma. If you do a good job, I'll give you a $20 tip. If you do a bad job, a kitten will die.
I'm disappointed this isn't another T5Gemma model designed for translation. The big use case I see for this is fine-tuning. What are people using this for?
Comparison to GPT-OSS-20B (irrespective of how you feel that model actually performs) doesn't fill me with confidence. Given GLM 4.7 seems like it could be competitive with Sonnet 4/4.5, I would have hoped that their flash model would run circles around GPT-OSS-120B. I do wish they would provide an Aider result for comparison. Aider may be saturated among SotA models, but it's not at this size.
Posts like this really mean "this doesn't work like I expect it to based on my background with some other technology".
But in this case, I tried in earnest to use nextjs for a project with auth & stripe, etc. this past week, and I can't believe how frustrating it is to get stupid things like modal dialogs to work properly in the client.
I have tons of experience with React SPAs. But the client/server divide in Next remains quite inscrutable to me to the extent that I'm just going to start again with Django (where I nearly started it in the first place).
So yes, it doesn't work like I expect it to either...
My day job involves training language models (mostly seq2seq) for low-resource languages (with substantially less data than 2GB of data).
A few thoughts:
1. You can't cut off the embedding layer or discard the tokenizer without throwing out the model you're starting with. The attention matrices are applied to and trained with the token embedding layer.
2. Basically the same thing regarding the tokenizer. If you need to add some tokens, that can be done (or you can repurpose existing tokens) if your script is unique (a problem I face periodically). But if you are initializing weights for new tokens, that means those tokens are untrained. So if you do that for all your data, you're training a new model.
3. The Gemma model series sounds like a good fit for your use case. I'm not confident about Hebrew support, let alone Hasidic Yiddish, but it is relatively multilingual (more so than many other open models). Being multilingual means that the odds are greater than they have tokens relevant to your corpus that have been trained towards an optimal point for your dataset.
4. If you can generate synthetic data with synonyms or POS tags, then great. But this is a language model, so you need to think how you can usefully teach it natural sequences of text (not how to tag nouns or identify synonyms - I also did a bunch of classic NLP, and it's depressing how irrelevant all that work is these days). I suspect that repurposing this data will not be worth it. So, if anything, I'd recommend doing that as a second pass.
5. Take a look at unsloth notebooks for training a gemma3 model and load up your data. I reckon it'll surprise you how effective these models are...
Just tested with a multilingual (bidi) English/Hebrew document.
The Hebrew output had no correspondence to the text whatsoever (in context, there was an English translation, and the Hebrew produced was a back-translation of that).
Their benchmark results are impressive, don't get me wrong. But I'm a little disappointed. I often read multilingual document scans in the humanities. Multilingual (and esp. bidi) OCR is challenging, and I'm always looking for a better solution for a side-project I'm working on (fixpdfs.com).
Also, I thought OCR implied that you could get bounding boxes for text (and reconstruct a text layer on a scan, for example). Am I wrong, or is this term just overloaded, now?
Is this because of the potential threat of a dollar squeeze? Seems like the market is too deep for that... Or is it just about the impact of volatility and the cost to the system of covering positions?
Why on earth are all the comments about her speaking fees? That paragraph was totally superfluous in the article.
What dictionary are you using?
[0] https://jcuenod.github.io/phrase-maze-poc/