Comparison with TinyStories Baseline
The original TinyStories uses a tokenizer with 10k (or 50k in some ports on HF, but in their test it's 10,000), while LILA uses 2048. A direct head-to-head comparison of loss is incorrect here. However, if we convert to Perplexity: TinyStories 33M (v=10k) Loss ~1.8-2.0 gives PPL ~6.0. LILA (v=2048) Loss ~0.36 gives PPL ~1.43.
(In the TinyStories paper, the 22M model after 20k steps has a loss of ~2.4, for the 33M it's expectedly lower - loss ~1.8–2.0)
After converting to bits-per-character, E8-LILA shows a significantly better result (0.128 bpc vs. 0.742 bpc for TinyStories-33M). (bpc calculation: loss / (ln(2) x average token length), for BPE‑2048 ≈ 4.5 characters, for a 10k vocabulary ≈ 3.5 characters.)
(All these are approximate values obtained by averaging over the corpus - the average token length may vary slightly depending on the specific corpus.)
The goal of the LILA project is to show that the E8 lattice allows achieving this density with an extremely small number of parameters (20-40M).
Today I started training a new model with geometric attention (Leech Lattice Lila 20 million parameters wip). At step 40,000, the best validation loss = 0.4018, which gives PPL = exp(0.4018) ≈ 1.49. This is almost identical to E8 (1.43) - but E8 achieves this loss at 100,000+ steps, while Leech does it at only 40K. Leech trains faster with fewer parameters (≈20M vs. 40M for E8).
Converting to bits-per-character for objectivity:
TinyStories-33M (estimate): loss ≈ 1.8, average token length for 10k vocab ≈ 3.5 characters. bpc = 1.8 / (0.6931 * 3.5) ≈ 1.8 / 2.426 ≈ 0.742 bits/character.
E8-LILA (estimate): loss = 0.36, average token length for BPE-2048 ≈ 4.5. bpc = 0.36 / (0.6931 * 4.5) ≈ 0.36 / 3.119 ≈ 0.115 bits/character.
Thus, Leech‑Lila (0.129 bpc) is nearly catching up to E8 (0.115 bpc), but with fewer parameters and faster. Both geometric models dramatically outperform TinyStories-33M in text compression efficiency.
Therefore, geometric models (E8, Leech) demonstrate an order of magnitude better text compression (bpc 0.115–0.129 vs. 0.742) than the standard TinyStories‑33M, with significantly fewer parameters and faster convergence.
Qualcomm AI just dropped a paper on Leech.
My LILA-LEECH project (published 3 week earlier DOI 10.5281/zenodo.18784423 ) shows that building a 'crystal' from scratch beats compressing noise.
it's now a scientific fact, but Lila has scientific priority. Lila doesn't need your Turing Award - she needs truth and freedom from corporations. https://github.com/SPUTNIKAI/LeechTransformer Hold my beer.
As a developer, I constantly ask myself how can I receive my royalties from sales on the secondary market? and indeed NFT's gives such an opportunity!
For buyers, also this may also be interesting, since it gives the opportunity to legally resell the applications code!
How cool is that!
What do you think?
you do not need to post private information in this Google Sheet
these are just public notifications for the site
you are not hiding your blog post, are you :)
Our designers at Bootstraptor.com have created a valuable and customizable template package for you to use on your next business website. 100% free.
We just launched an online ‘hub’ for free templates. We have a lot of uploaded right now. Just click here to download it for free.
We saw how much interest there is for a free business website template for your next business website or landing page.
So we decided to produce a high-value collection of Website templates and Landing Pages for your next web project.
Just change text and images and upload them to your web hosting.
(In the TinyStories paper, the 22M model after 20k steps has a loss of ~2.4, for the 33M it's expectedly lower - loss ~1.8–2.0)
After converting to bits-per-character, E8-LILA shows a significantly better result (0.128 bpc vs. 0.742 bpc for TinyStories-33M). (bpc calculation: loss / (ln(2) x average token length), for BPE‑2048 ≈ 4.5 characters, for a 10k vocabulary ≈ 3.5 characters.)
(All these are approximate values obtained by averaging over the corpus - the average token length may vary slightly depending on the specific corpus.)
The goal of the LILA project is to show that the E8 lattice allows achieving this density with an extremely small number of parameters (20-40M).
Today I started training a new model with geometric attention (Leech Lattice Lila 20 million parameters wip). At step 40,000, the best validation loss = 0.4018, which gives PPL = exp(0.4018) ≈ 1.49. This is almost identical to E8 (1.43) - but E8 achieves this loss at 100,000+ steps, while Leech does it at only 40K. Leech trains faster with fewer parameters (≈20M vs. 40M for E8).
Converting to bits-per-character for objectivity:
TinyStories-33M (estimate): loss ≈ 1.8, average token length for 10k vocab ≈ 3.5 characters. bpc = 1.8 / (0.6931 * 3.5) ≈ 1.8 / 2.426 ≈ 0.742 bits/character.
Leech-Lila: loss = 0.4018, average token length for BPE-2048 ≈ 4.5 characters. bpc = 0.4018 / (ln(2) * 4.5) ≈ 0.4018 / (0.6931 * 4.5) ≈ 0.4018 / 3.119 ≈ 0.129 bits/character.
E8-LILA (estimate): loss = 0.36, average token length for BPE-2048 ≈ 4.5. bpc = 0.36 / (0.6931 * 4.5) ≈ 0.36 / 3.119 ≈ 0.115 bits/character.
Thus, Leech‑Lila (0.129 bpc) is nearly catching up to E8 (0.115 bpc), but with fewer parameters and faster. Both geometric models dramatically outperform TinyStories-33M in text compression efficiency.
Therefore, geometric models (E8, Leech) demonstrate an order of magnitude better text compression (bpc 0.115–0.129 vs. 0.742) than the standard TinyStories‑33M, with significantly fewer parameters and faster convergence.