The 16MB constraint is a fascinating forcing function.
Most architectural improvements in recent years have come
from scaling, so it'll be interesting to see whether
depth recurrence or aggressive parameter tying can
meaningfully close the gap.
Curious if anyone has a prior on how much bits-per-byte
can realistically improve over a well-tuned baseline
at this parameter count.
Curious if anyone has a prior on how much bits-per-byte can realistically improve over a well-tuned baseline at this parameter count.