I've read the entire series of Culture novels and don't recall seeing "starmind" as a term anywhere. Mind, yes, but used in a somewhat different context, as the minds are both sentient conversational AI entities with equal or greater intellect to a meat-based human or alien, and also semi-godlike AI powers (a single Mind has the capacity to have a 1:1 conversation with all of the residents of an Orbital if it wants to).
I'm curious, where are you seeing M.2 2230 to M.2 2280 size NVME SSE that exceed 4.5 to 5GB/s sequential reads for large files such as a GGUF (likefrom an ordinary ext4fs file system with default options)? The PCI-E 4.0 or 5.0 bus they're attached to might be capable of greater speeds, but the bottleneck is the flash and the flash controller.
The benchmarks I'm seeing for many of them don't really make me think that a pair of consumer grade NVME SSD you could fit in a mini-PC or mini-itx size desktop would, added together, be capable of 20GB/s reads.
For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.
The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.
The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.
Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.
> So space considerations were very real when a large part of your customer base existed on 300 & 1200bps modems in the mid 90’s.
I somewhat did a double take at this.
My family had by no means a really generous budget for i386 platform home PC equipment in the early to mid 1990s, but I clearly remember a basic ISA interface, internal, hayes compatible 2400 baud modem in mid 1993 being really affordable and fairly shortly thereafter getting upgraded to a 14.4k. The 2400 bps modem came as part of a package deal with a 486DX/33 we bought in spring 1993.
In the local BBS scene in mid to late 1993 there were pretty much zero people actually using 300 or 1200 bps modems.
It might have been different if you went back to like 1990 or mid 1991, probably there were a lot of people with 1200 baud modems, but that would be before my time.
If we say that "mid 1990s" means 1994-1996, by that time almost everyone had upgraded to 14.4k and then very quickly to 28.8 modems if they could afford it, by mid '96.
I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.
I was just using that as an example of constant on going price rises, it was the most mundane and not particularly fast ddr5 6000 stuff. The 6400 is even more ridiculous.
I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.
Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
Most any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue.
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Q8 isn't overkill if you have sufficient RAM to fit the whole model, and you care about quality. There's a number of people who have enough hardware to fit exactly one 27B to 35B size Q8 model and not more than that, so if you can fit the whole thing in Q8, no reason to use Q4 or Q6.
At least on Android there's a fair number of apps like "osmand" which can download an entire state, province or country vector/map data, in their own semi-proprietary data format derived from public openstreetmap, for truly offline use anywhere you have a GPS signal. It'll even work on a phone that has no SIM card in it.
MDM is designed for corporate owned phone environments where there's a great many good reasons to lock down a phone. If you're handing out company owned phones to employees you want the ability to remotely lock/wipe, install and remove apps, set a number of restrictions. If people want to do anything else they are completely welcome to do it on their own personally owned property.
For instance I have recently seen a very successful Apple MDM deployment in a school environment where the teachers and staff have access to a great depth and breadth of PII of a thousand children under age 16. You don't want all those phones to become a free-for-all of people doing whatever they want.
I think if you had something like a theoretical used/refurb 2U rackmount server with two older multi core CPUs, 768GB of RAM, you would see faster performance loading a Q6 or Q8 GGUF of GLM5.2 into a freshly-compiled latest copy of llama-server, with the "no-mmap" option turned on to intentionally load the whole thing into RAM at the time the llama-server daemon launches.
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
Another thing that seems to be disturbingly common these days is some party involved in residential real estate transactions having a RAT on one or more PCs, and/or compromised email account, intercepting an email at the closing stages of a house purchase and sending someone information to send a wire transfer to a fraudulent location.
I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".