Cloud models can use batch processing which is significantly more efficient. A local model has basically a batch of one which takes as much time to process as a batch of 100 because the gpu is memory bound and spend most of its time loading the model from vram to the gpu cache while the gpu cores are idle. With a batch of 100 the model loading time and compute time are roughly similar. So local
Models have a first 100x lower efficiency. Secondly, local models are idle most of the time waiting for the user to write a prompt, so the efficiency gap is probably more around 1000x.
I have done that at meta/FAIR and it is published in the Llama 3 paper.
You usually start from a seed. It can be a randomly picked piece of website/code/image/table of contents/user generated data, and you prompt the model to generate data related to that seed.
After, you also need to pass the generated data through a series of verifiers to ensure quality.
For the knowledge preservation, I guess that a copy of deepseek has most of the required information. But, it would be hard to run it in a primitive world.
But we have landfills which are full of great raw materials. I would argue that it is easier to collect steel from a landfill than from a mine during the industrial revolution.