The problem is if DDR4 was viable here it would also be absurdly expensive. There’s no world in which we survive this demand without a corresponding increase in supply. It’s not like there are zettabytes of spare cheap DDR4 sitting around.
I’ve got a 128gb m5 max mbp and two sparks. For my real-world use cases, a single spark running DS4 Flash will have fully responded by the time my Mac has even started generating tokens. I figured I’d have more generation heavy work when I also bought the Mac, but it has done very little work running LLMs since I got the first Spark.
I mostly run them clustered for DS4 and am quite happy with the performance, and the cost isn’t that much more for two than the MBP while giving me double the unified memory.
I’ll probably pick up a third to run multiple smaller models. I don’t understand why people would buy a halo over a spark at comparable prices, particularly because if you want to cluster, the cx7 be beats the shit out of them when it comes to latency and throughput
Anything where you're dealing with a large volume of records/documents. Lots of people are using these for large-scale digitization of documents - scanned stuff being OCR'ed and summarized, generating embeddings, etc. Large scale translation.
Anywhere where you might have a large backlog of data to work with can end up in this sort of situation.
The dgx spark is the same chip and those are in the low 3s to 5 range for most of them depending on manu, storage config, etc. The dgx sparks also have connectx 7 cards in them to support the 200gbps networking for RoCE.
So I would expect the mini PCs to come in less than the sparks. Laptops I assume will be close in price with the addition of all the other laptop stuff.
Prefill is another advantage vs. Apple. It's way way way way faster on a spark than it is even on an m5 max.
Same model, same quant, same query, as close to as matched settings as I can get from vllm, and for workloads with large prompts + low cacheability, one of my sparks will often be done responding before the mbp is done with prefill.
Not true. This is aimed squarely at the Strix Halo and Mac markets. It's basically just strictly better than the Strix, and it's not clear cut vs that Macs in any sort of blanket statement.
My M5 Max 128gb MBP decodes faster than one of my Sparks, but the Spark's prefill is so much faster it can often answer the same query before the mac's prefill is finished. If you have large prompts, low cacheability, etc., a spark might be a very good options.
Not to mention you get can get two sparks and the MBP will be 85%+ of the cost at half the RAM.
I'm kind of tempted to pick one up. Leave running big models to my dual dgx setup, and all the misc. random stuff on an rtx.
For these in specific, they appear basically transparently to the GPU. There's a lot of software/firmware stuff for this, but also a different hardware architecture - while the RAM is on the CPU die, the nvlink-c2c gives it extremely low latency and 600GB/s bandwidth between the GPU and CPU.
Which is part of the problem with all of this nonsense right now - everyone is running off of emotion and not looking to see if what is being said is actually true. Which is somewhat ironic, considering the message of the article.
Never had this issue with any of my MBP. I never even have magsafe available, couldn't tell you where a USB-C to magsafe cable is in my house if my life depended on it.
There are a variety of inference engines that support this, regardless of whether or not there is native FP8 in Ampere - llama.cpp will do it quite happily. VLLM you can do W8A16 quant too.
There are a whole lot of ways to quantize models in general.
I remember getting my first CalDigit TB dock and being excited - everyone seemed to love them. I expected it to largely Just Work.
That thing Didn't Work more than it Worked, but options were slim. Eventually it fully died about 14 months in. I didn't even bother checking to see what the warranty terms were. TS3 Plus, back in 17 or 18. What a piece of shit.
Sounds like it's a good thing I didn't bother trying again in the early 2020s and only recently bought a new dock.
I've had multiple X520-DAs in desktop cases for years without doing anything special for cooling. Hell, one of them was in a fully watercooled system that had very little in-case airflow.
To the best of my knowledge, the Indian support teams for AWS have all the same access their US/EU/etc. support teams have when it comes to escalation and communication avenues. Obviously real time communication is harder for them than the US counterparts speaking to teams in Seattle, but the same is true for a team in Ireland or Australia.
If you are building a follow-the-sun support organization for a technology company, would you really avoid building a support team in India? Seems like it's the country most likely to have a large enough number of people with the expertise and language skills to staff for the need, even if you were paying them the same as their US counterparts.
From talking to people I know on support teams at hyperscalers and other tech companies: A mind-boggling huge portion.
But even as an AI believer and thinking it's great for this sort of thing, I also think if it's good enough for this sort of work, it should be good enough to identify when it's not capable of providing value, too.
You have a minority view on this argument, though. Scientific and structural realism both reject the idea that math is just a map. You've got company with the instrumentalists and antirealists, but the majority consensus is that math is somewhere between the structure underlying the territory to all the territory.
Zero was already part of the territory. Lack of something is a very normal state in the universe. Once we added it to our understanding of math, we were discovering it, not creating it. Of course people who are scientific or structural realists would agree it didn't change reality - because reality already had it, whether we knew it or not.
Well, I was thinking more along the lines of, say, multiplication and division - you can handle every single equation humanity has ever come up with without either of them. It might be messy and awful and annoying, but I would say in particular these operations are invented more than discovered.
So, more properly phrased, we created some operations.