This made me wonder about a 3D printer alternative that builds things by folding a thin sheet of metal into arbitrary shapes instead of extruding filament.
Even if you had all that you would get completely different weights out at the end, and you also don't have the resources to "compile" an LLM because the compilation can cost $100M. If you were given the training data but not the weights, would you consider that open source?
This can't be used to save VRAM in practice. To generate a new token with the primary model, you first need to decompress the cache, which involves regenerating the whole sequence from scratch. I.e. generate 1 million tokens with the small model to generate 1 with the large.
You can use the original model to compress the kv cache and get ∞x compression, since the prediction is perfect. The cost is time, and I don't see how this could be worth it.