for the on-device model, yes it runs on the Neural Engine (at the moment) so a newer chip means faster, cheaper local inference. For the server side path this Claude package is about your machine is irrelevant since it's a network call. The same API covers both, so "best machine for AI" only bites when the session is actually local.
But we can imagine that the balance of what's on-device vs what's remote will move continuously towards the former as time, improved HW and improved local models keep progressing
it's cloud, the doc is explicit that requests go straight to api.anthropic.com with Apple not in the way.
so Claude via FM dies offline while Apple's on-device SystemLanguageModel (the ~3B one) keeps working. It isn't a hybrid really: the framework just has both implement the same LanguageModelSession protocol so "local 3B" and "remote frontier model" become a one-argument swap.
IMHO what's worth internalising is that the two share an API but nothing else: the on-device path runs on Apple's Neural Engine and costs battery (you can watch ANE power ramp while it works) while the cloud path costs API credits/tokens and does zero local compute. Same code, opposite cost model.
something I haven't seen highlighted anywhere yet, while I find it very interesting, is the distributed inference across Macs (JACCL over Thunderbolt 5), an OpenAI-compatible mlx_lm.server, agentic-on-Mac.
Apple keeps MLX (bring-your-own-weights) separate from Foundation Models / Core AI.
among the many things I found that were not highlighted anywhere (that I could find) is the distributed inference across Macs (JACCL over Thunderbolt 5), an OpenAI-compatible mlx_lm.server, agentic-on-Mac.
Apple keeps MLX (bring-your-own-weights) separate from Foundation Models / Core AI.
lol, it took me 48 hours to do (and re-do, and re-do) this test + write it up and now that I convinced myself to stop changing bits and just publish it... Google's just announced the Gemma 4 QAT models :-D
It would not change the core of my article since the bottleneck remains the memory bandwidth on the old M1 16GB though
not obvious at all when every sentence uses "you" to indicate a general rule that applies to every Italian rather than "I" to indicate a personal experience
there is no lost love between me and Wall Street "bros" BUT it's not their job to run the companies, their job is to buy and sell shares based on results and potential future results.
When a company decides to go public and take their money it's still the company's responsibility to run their business and keep the people they took money from in exchange for shares happy.
as usual it's a matter of bubbles: most of those you know might hold such resentment, most of those I know are in fact in love with the possibilities and are trying their hardest to leverage the new tools.
My 64yo "non digital" graphic artist aunt holds a very high level of resentment to the digital ones, while most of her old friends and ex-colleagues who embraced digital way back when are stil active in the space and happy one way or another, she is not.
But she is happy she can now get near-real-time two-way translation to/from languages she doesn't speak and is also happy to bury her head in the sand when I point out that's thanks to the same tech that will have an impact on the people who do simultaneous translations as a job.