MoEs don't route like most people imagine. They aren't learning topic based experts despite the name
The original Mixtral paper [0] (in the "Routing analysis" section) found:
"surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic"
A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing).
> no other widely accessible path for young people to build a future
This is exactly right. Widespread gambling has a fundamental nihilism baked into it. If you believe hard work and frugality are the best path to success, then gambling doesn't feel like a good use of your income and energy.
But if you feel like your dreams are falling out of reach despite your hard work, if it starts to look like even big investors are really just playing a game (often with loaded dice) rather than acting on fundamentals, if grift seems like a more reliable path to success then hard work and ingenuity, then all of a sudden gambling seems a lot more sensible. Even if the odds aren't in your favor, you can at least get lucky when you gamble. When the game feels rigged its easy to believe that hardwork is destined for failure, whereas even loaded dice have some probability of landing in your favor.
If you're using a MoE model, then why do you care about the larger RAM offered by these devices? That's the main problem with low bandwidth devices: they limit the effective ram you can make use.
I do (and have historically done) quite a work with both local LLMs and local diffusion models. I have an M3 Max MBP at 400 GB/s and also a desktop with a RTX 4090 with 1,008 GB/s
While the M3 Max MBP can serve up MoE reasonably fast (~60 token/sec)the RTX 4090 is an entirely different experience (~170 token/sec). I also do a fair bit of experimentation and am currently running a custom decoder that requires expensive look-ahead, but I'm still able to get a usable 25 token/s on the RTX.
The raison d'etre for the DGX spark is not practical home inference, but rather offering the same fundamental architecture as data center cards for a affordable CUDA prototyping. If you want to build software to run on H100s, you probably can't justify buying (and running) a single card. The DGX spark solves this by having the same fundamental setup as what those cards have.
That makes these non-NVIDIA DGX-like devices confusing to me. The entire benefit of the DGX series is the NVIDIA architecture itself.
Anyone interested in home LLMs should decide whether a Mac or a dedicated GPU is the more sensible path based on their budget and other computer use. Each has their own benefits.
> Yet those 2-3M channels get the lions share of the views.
...because they're directly tied to YouTube own revenue.
As someone who creates a fair bit of content, and am fortunate enough not to have to worry about the revenue from that, I deliberately try to minimize monetization because it ultimately becomes the main driving force behind your content creation.
Monetization as the only model for content creation leads to an incredibly safe and boring world while simultaneously meaning every piece of content you interact with is trying to extract money from you the viewer.
In general if you're setting up a local LLM you should assume it's going to be primarily working as a server and talking to various clients. I use my MBP, but that's because I don't travel much anymore so it can happily work as a server at all times. With the right agent setup you can probably manage most things from your phone even if you don't have a seperate machine to use as a client.
I have an older laptop I run a hermes agent on backed by an API based open (non-local) model and Macbook Pro M4 for running another model locally (also using hermes). The agents have a Mattermost (open source version of slack) server they run and I run Mattermost on my phone so I can talk to them and task them with things. In fact, it was through the hermes WhatsApp endpoint that I got the first agent (non-local) to setup the Mattermost server and unboard the second agent (local mbp).
Then I can just chat with them through Mattermost when I need work done. Whenever I need something done I just hope on the Mattermost server and chat with them. I've had them build me multiple research reports (the fully local agent did awesome at this), learn how to use Stable Diffusion on my desktop to generate images, install and perform maintenance on various local services I run (including Open WebUI).
It depends on your use case. There's a lot of hype around machines like the DGX spark (I'm assuming this is the type of device you're referring to) because they look awesome, and are priced reasonably well. However all of these have notoriously low memory bandwidth despite the high ram.
These devices, especially the DGX line, are fantastic if you are interested in low-level CUDA programming. The DGX spark can be used to prototype CUDA code/libraries for GPUs that most of us couldn't think about affording. If you want to learn how to program for datacenter level GPUs then these are the best way to get that at home. Sure your code will run very slow compared to the real thing, but you can take that code and, theoretically, run it on the real thing. For anything else though, I feel there are better options.
If you're interested in pure inference I'm pretty partial to Apple devices. The M4 Max gets you 546 GB/s, the M5 MAX 614 GB/s, and the M3 ultra (you'd have to buy used at this point) 819 GB/s. Plus you have a very useful computer even if you realize you don't want a full time home inference server. Additionally these devices require very low power (if you're running high end consumer GPUs you do have to think about what your energy costs are per hour and how warm you like your room).
If you're interested inference and training, or already have a pretty beefy desktop PC, or simply demand the most token/s you can get, then GPUs are the way to go. The downside is they're still pretty memory restricted (but honestly the options for what you can run on any RTX N090 are pretty good). You'll get blazing inference and prefill speeds on these devices. The only down side is, if you are using them heavily, you will see it on your energy bill and feel it in your room.
The "should I wait" question is also potentially applicable. The world of consumer hardware is looking increasingly bleak (and expensive) but if Apple does release a new "Ultra" model we could be looking at inference speeds very close to GPUs (there's still limitations to these devices that makes training preferable on GPU)
My experience working in the open model space pretty deeply (both LLMs and diffusion models) for years now is that it is not quite as simple as that.
In the open model space an insane amount of effort goes into getting more powerful models to run with the same or less RAM. For example in the diffusion world many things that could not be run on easily under 24GB of VRAM actually run much better today with much less VRAM than they did a few years ago. You can do many things today with 8-16GB of VRAM that would not have been possible. At the same time the most advanced open models, like LTX 2.3 for video gen, still seem to respect 24GB of VRAM as the upper bound.
Similarly the standard "big" but localish open model for LLMs back in the day was Llama 3 70B, this was both a much worse and much larger model than Qwen 3.6 27B
So in two different spaces I've witnessed the "RAM required to run the best" decreasing or at least remaining stable, while the performance being achieved in both areas is astounding (LTX 2.3 is faster, better and more capable than the Wan 2.2 model that held popularity before it).
The biggest thing to watch out for is not just RAM/VRAM but memory bandwidth. You can try to "future proof" yourself with lots of RAM, but if it's 400 GB/S you're still constrained to smaller models.
In addition to models getting better, the quantization methods have also got much better. If you already have an RTX 3080 it's absolutely worth the time to just mess around and see how it does, experiment with different quants that fit in your VRAM. If you're purchasing I would recommend coughing up the extra cash for the 3090.
If you are experimenting it's worth mentioning that the harness/tooling is very important to getting a solid experience. Herme's agent is great for running helpful agents and OpenWeb UI can get really make the experience feel on par with paid chat interfaced.
A reasonable halfway step is to pay for an open model through the provider or open router. You'll get many of the benefits (especially around pricing) without needing to shell out on hardware before deciding if you like the way these models work.
I don't have the number around but there is a notable latency for pre-fill on the M3, but once it's running the delay is negligible.
The RTX, unsurprisingly, is all around superior performance wise, but: I use that computer for gaming and image gen work so I can't dedicate it as a server, and, especially when it's warmer, the heat generated under heavy loads is noticable.
There are a couple of things, but basically it boils down to the same reason people prefer Linux to Windows/MacOs: customization, control and privacy (arguably all of these are really subsets of 'control').
Having full control over how your data is retained, what the system prompt is, which version of the model you're running, etc leads to much a more consistent experience. For example, for chat sessions, I can't stand the new "let me push back" version of Claude. For my home models I never have to worry about that.
There's never a mystery as to whether the model secretly degraded performance, I always know exactly which model I'm using and how well it's utilizing resources etc. Open models also give you full visibility into the reasoning steps, so you never have to guess what the model is thinking.
Then when you start getting into things like uncensored/abliterated models we're talking about something you can't even pay for. In case you're unfamiliar, even open local models have guardrails built in. But people in the community have found ways to remove these. One of the things I've found most concerning about AI, which is under discussed, is the combination of people having personal chats with an agent that both monitors the conversation and refuses to discuss certain topics. This leads to a very deep level of self-censoring I find dystopian.
I also have multiple hermes agents setup, some with local backends other with open but non-local backends (e.g. Kimi through the API). For some tasks, I've just started to find the local agent tends to work better for the type of tasks I want (maybe it just over thinks less?). I don't use it for coding so much as research tasks and sysadmin stuff, but I've been really happy with the results.
Oh, and let's not forget, especially running on a Mac, these local models are basically free to run.
See my comment to parent. I've been using local LLMs for practical, personal tasks for a few months now very successfuly.
You can run fantastic local models if you have either:
- M-series Apple device with ideally >= 24GB of VRAM
- RTX [345]090 GPU
I'm fortunate enough to have both and use an M-series laptop as basically a persistent server (I don't use it much and when traveling typically just use my work laptop). My desktop doesn't act as a persitent server but I fire up llama.cpp on it all time for quick chat sessions.
If you have one of the above devices and can dedicate it as server there are additional layers of tooling you can use that dramatically improve the experience. In particular Open WebUI allows you to add tons of useful tools (image gen, web search, code eval, etc), and agent harnesses like Hermes can make the current gen small models very capable. I have an agent in chat on my phone that basically handles all the sys-admin for the server it runs on.
I have a home server that runs Qwen3.6-35B-A3B through llama.cpp with Open WebUI for the user facing interface.
My teen isn't super interested in AI, but whenever they do feel curious they have their own account they can use on our home network. As far as chatting goes local models are more than capable for handling standard chat questions, doing research, helping troubleshoot problems etc. In fact it was an agent powered by the same model that setup the open webui server and took care of all the account management features through my phone (using Hermes agent).
If you're building AI powered features and using sophisticated agent setups for coding for work, then it make sense to use SoTA from these providers. But I've been using local models increasingly for personal use and am starting to find them preferable (I run an uncensored, ephemeral model for my own use and it's an entirely different experience than anything you can pay for).
Still haven't cancelled my personal Anthropic subscription, but considering it soon.
> It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time.
In the monkey example the infinite time is doing a lot of work there. The fact that LLMs can search through semantic space and find reasonably correct paths in a reasonable time is directly tied to the reason why they are valuable.
Saying "these two things are similar except one can be useful and one can't" is not a great comparison.
For me the real lesson learned isn't how "smart" LLMs are, but rather how much human work is basically reducible to repeating past work with minor variation. Human's believe they are "reasoning" but so much code writen is just the human brain doing the same autocomplete style work that LLMs can do now.
I used to think comparisons of AI with the web where ridiculous, but increasingly it looks like they're not that dissimilar as far as how they change how we work. But as someone who graduated college during the bust there was a loooong gap between peak hype and things like online banking and e-commerce becoming standard.
Even if AI is an absolutely bubble, and SpaceX, Anthropic and OpenAI all cease to exist in a year... there's simply no way that AI has not fundamentally changed work. Even if I was forever pinned to the local models I'm running and the agent harnesses they use, I would never write code for work the same way.
But I lived through the rise of the web. I remember serving dynamic websites through cgi (which meant a new instance of an interpreter was spawned per user session). I vividly remember great JavaScript books saying things like "never use JavaScript for core functionality". I recall Java engineers saying Ruby on Rails was a toy and would never take off, that Python offered nothing over Perl and that "rich web applications" where never going to replace app native interfaces. I remember when the MVC pattern from the early Smalltalk days being dusted off and repurposed for web applications, completely changing how we designed software for the web.
And all of that is just software. It wasn't until the pandemic that ebooks replaced print books in share of academic library circulation (reversing a decades long trend of reduced circulation).
In my daily use of agents for coding and other forms of problem solving, while it is a wild accelerant, it's also clear we have not even started scratching the surface of how to think about building things with these tools.
I suspect we'll adapt to AI faster, but having lived through one major tech revolution, transforming work still takes some time. I'm not surprised we don't see an immediate jobs crisis.
Not to mention the completely separate topic that huge classes of employees were not and are not all that productive, so boosts in productivity don't imply lost jobs. That would require a boost in productivity combined with pressure to create concrete value with less, looking at the SpaceX IPO we're still a ways away from working about how efficiently we create concrete value.
It's more insidious than that. These IPOs aren't being rushed, they were waiting for all the pieces to be in place to force 401ks and other retirement plans to buy these IPOs.
The most recent change was the NASDAQ adopting the "fast change rule" which allows newly IPO'd companies to be listed in the index after only 15 days of trading. This rule was decided March 30, 2026 and only came into effect May 1, 2026.
The plan is to rapidly drive these prices up in the first 15 days, get the companies listed in the NASDAQ so funds are forced to purchase them at higher prices, then leave retirement accounts holding the bag.
As you likely know, rules have recently been changed that basically force many 401k funds to invest in these IPOs while simultaneously having a relatively small number of the initial IPO to be sold to the public forcing the funds to by at inflated prices.
The bubble won't pop until these retirement accounts of have been raided.
Have you personally used any of the latest batch of even smaller local models? They certainly don't beat SotA models at coding... but with a good harness they are able to achieve things with SotA that I couldn't last year.
I've repeatedly given local models non-trivial projects that involve research and coding which they've successfully completed with minimal intervention from me (almost exclusively in the domain of reviewing the results). Again, nothing comparable with current SotA, but definitely tasks I could not have given SotA models last year (without agent harness).
Now that pure progress from these models seems to have slowed down, we're seeing a ton of options for both making models more efficient and other tools that help improve them (everything from agent harnesses to RLVR).
That's just looking at "what can small do today", when you look at what's possible with larger open models that are still much smaller than SotA from the major providers, their performance is extremely close to SotA, enough that for personal projects I'll just use Kimi instead of any anthropic offerings.
So it's not terribly hard to image a solution in the middle happening within a few years. We still have tons to learn about optimal sizes of these models and how to build them with maximal efficiency (and we've already seen a lot of recent improvements in this space).
The original Mixtral paper [0] (in the "Routing analysis" section) found:
"surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic"
A quick skim of more recent analysis on MoE shows that this hasn't changed. MoE models do appear to work, but don't appear to do what the name implies, if anything they're routing based on the structure of the text and not the semantic content (and we're still not entirely sure what they're doing).
0. https://arxiv.org/pdf/2401.04088