The claim hinges entirely on the narrow definition of "large-scale energy generation." For the UK, with its high seasonal energy demand and low winter solar output, the cost of generation is almost irrelevant next to the cost of firming that power for 24/7/365 availability. While the paper[1] shows solar PV and daily-cycle batteries are getting cheaper, it also shows seasonal storage solutions like hydrogen are still an order of magnitude too expensive and inefficient (huge capex for electrolyzers/storage + poor round-trip efficiency). So providing reliable, 24/7/365 baseload power from PV + storage in the UK is demonstrably not cheaper than gas or nuclear today.
> At the time of writing, the highest-ranking non-purely-transformer-based model on the LM Arena is Jamba, which is a transformer–state space model hybrid, at rank 96.)
From the paper, section 2.1:
minimize_θ,φ,Δ ||P_φ(Δ, E_θ(x)) - sg(E_θ'(y))||_1
where
y - full video, x - masked video, E_θ(.) - learned encoder (semantic embedding), P_φ(.) - learned predictor, Δ - learned mask (which patches in a video where dropped), sg(.) - stop gradient to prevent change, gradient propagation in E_θ'(.), which in turn is an exponential moving average of E_θ(.) ie. θ'_new <- τ θ'_old + (1-τ) θ. So the loss is applied only to the predictions of the masked patches while the encoder of full video follows the learned one. This asymmetry in learning prevents collapse of the encoder to a trivial constant.
I was thinking, to build a large, efficient factory autonomously, one could use LLM as a high level agent that is using specialized tools. The overall strategy would perhaps look like following:
1. create a (intermittent) goal for a resource production
2. create a factory graph with calculated number of machines and number of resources required to transport between them. This would be done by using linear programming (factorio calculator)
3. somehow map the resulting graph to a hardware description language. Such that each entity would be mapped to unique logic component. And each transport lane would be mapped to a unique wire (most difficult)
4. compile to 2d FPGA layout using all the VLSI algos like partitioning, routing (hdl compiler)
5. map the resulting plan back to a concrete factorio design
This doesn't apply to energy grid scale projects. Also that is why energy sector in most countries is privatized. Take Germany with total energy costs that would be like ~10% of the federal budget (about 40 billion euros). Making energy even more expensive would only lead to further deindustrialization and lower gdp making it politically non viable. Also, you can't just throw money and buy storage like with typical government contracts - it requires building entire new industries (like Germany's struggles with hydrogen infrastructure).
none of those storage solutions are easy for grid scale. For batteries you need whole factories, pumped requires permits and tone of cement, chemical is like hydrogen with low efficiency. All of those are expensive which for government is synonymous with difficult.
Solar PV operates 10-30% of the time (depending on location). Without storage, this naturally limits direct solar contribution to about 20% of energy demand. Going beyond requires expensive storage solutions.
[1] https://www.authorea.com/users/960972/articles/1329770/maste...