> Every corporate network to which I've connected worked just fine without it.
Just because it appears to be working fine doesn't mean you are in control of it. Without hardware attestation, how do you know the machines are running the software you think they are?
If regular people can repurpose old hardware, so can shared providers, who can extract more value from the hardware and thus afford to pay more.
In a constrained market, supply and demand favors folks who can most efficiently extract rent. Local models only make sense in a world with abundant compute and energy.
My point is that dram demand is mostly orthogonal to whether everyone is using open weight models or secret weight models. Heavy demand for local models (whether secret or open weight) will require even more aggregate DRAM than for shared.
Demand will only go down if people reduce their use of these AI tools. Given how much folks here complain about quotas, I'm very skeptical that will happen willingly.
There isn't enough hardware in the world for everyone to run their own SoTA model. The only hope we have is if we work together to host these on shared infrastructure, benefiting from >50x economies scale due to batching, etc. That infrastructure doesn't have to be owned by greedy corporations.
That DRAM would get even more use if it was removed from these machines and placed into a shared pool :)
I joke, but thanks to the brutal DRAM market there has been some movement in this direction lately...
My point is that it is WAY more efficient if we put the world's DRAM supply into a shared inference pool instead of stranding it in local machines where it won't have as high of batch size or utilization.
The cost of not being efficient is even higher DRAM costs than we have now, given supply and demand.
I think folks in this thread are underestimating how expensive it is to serve a SoTA model at 100 tokens a second. In addition to the $500k in capital costs, you also have significant electricity costs.
This stuff is expensive because supply is much lower than demand. If everyone was to run their own hardware with a batch size of
1, we'd have 100x more demand for inference hardware and electricity than we do now, and people would be even more frustrated. Efficiency is everything, and we need all the economies of scale we can get to meet demand.
Good luck finding reasonably priced switches and low power PD ICs that support type 3 or type 4 PoE.
Also, supporting those tiny pulses requires large capacitors to hold a charge in between pulses. That plus the required magnetics make PoE sensors way more bulky and expensive to manufacture than old fashioned RS-485 sensors.
PoE is lousy for sensors. The switch will cut the power if you draw less than 10mA (480 mW), so regardless of PHY efficiency (which is terrible compared to most RS-485, CAN, or even radio ICs), you are REQUIRED by the spec to generate heat that will mess up your sensor measurements.
I remember using this thing when I was a kid, trying to figure out how all the switching effects worked, so stumbling on this manual many years later was really satisfying...
I had the misfortune of writing a complicated WPF app from scratch circa 2010-2011. Performance using the WPF widgets was terrible compared to HTML/Javascript/Blink; we ended throwing away most of the WPF code other than the main shell and a few dialogs, reimplementing the importantant stuff with immediate-mode Direct3D/Direct2D to get the necessary speed.
I recall wasting a lot of time staring at decompiled .NET bytecode trying to understand how to work around many problems with it, and it was clear from the decompiler output that WPF's architecture was awful...
Most software uses 10x more memory than is necessary to solve the problem. In an ideal world, developers would stop building bloatware if their customers can't afford the DRAM.
At the end of the day, I'm being paid to ensure that the code deployed to production meets a particular bar of quality. Regardless of whether I'm reviewing code or writing it, If I let a commit be merged, I have to be convinced that it is a net positive to the codebase.
People having easy access to LLMs makes this job much harder. LLMs can create what looks at the surface like expert-written code, but suffers from below-the-surface issues that will reveal themselves as intermittent issues or subtle bugs after being deployed.
Inexperienced devs create huge commits full of such code, and then expect me to waste an entire day searching for such issues, which is miserable.
If the models don't improve significantly in the future, I expect that most high-stakes software teams will fire all the inexperienced devs and have super-experienced engineers work with the bots directly.