Could anyone suggest a tool which would take a bunch of PDFs (already OCR-d with Finereader), and replace the OCR overlay on all of them, maintaining the positions? I would like to have more accurate search over my document archive.
It can be relatively cheap too under the constraints imposed by typical AI workloads, at least when it comes to getting to a 1TB/s or so. All you need is high-spec DDR5 and _a ton_ of memory channels in your SOC. During transformer inference you will easily be able to use those parallel, multichannel reads. I get why you'd need HBM and several TB/s of memory bandwidth for extremely memory intensive training workloads. But for inference 1TB/s gives you a lot to work with (especially if your model is a MoE), and it doesn't have to be ultra expensive.
What's frustrating is there's no real reason why regular DDR5 can't reach 1TB/sec with a sufficient number of channels. The manufacturers are just holding it back to drip feed that memory bandwidth over several generations. Except for Apple, which lets you have 800GB/sec now, and will let you have 1TB/sec+ in M5 Ultra next year. It's $$$$, but still - the true alternatives are much less cost effective.
With what memory bandwidth? Remember, without e.g. speculative decoding you need to read the entire model and KV cache for every token. Let's be extremely generous and say you get 512GB/sec in memory bandwidth, on par with a high end M4 MacBook Pro. This means you can only read the entire DRAM 4 times a second, generating at most 4 tokens per second. Smaller models will of course run proportionally faster, but 128GB isn't by itself a sufficient statistic to say whether this is "theoretically awesome" or not.
The best beginner assembly hands down is Z80. It may be a bit harder to start with, but if I could write games in it at 14 years old, an adult should have no issue at all.
My models are both 4 bit. But yeah, that could be - small models are much worse at tolerating quantization. That's why people use LoRA to recover the accuracy somewhat even if they don't need domain adaptation.
This is from a small model. 32B and 70B answer this correctly. "Arrowroot" too. Interestingly, 32B's "thinking" is a lot shorter and it seems to be more "sure". Could be because it's based on Qwen rather than LLaMA.
If they just wanted to win, they wouldn't be offering these models for free. Meta is also trying to salt the earth for the other players. "Commoditize your complements".
Downloaded the 14B, 32B, and 70B variants to my Ollama instance. All three are very impressive, subjectively much more capable than QwQ. 70B especially, unsurprisingly. Gave it some coding problems, even 14B did a pretty good job. I wish I could collapse the "thinking" section in Open-WebUI, and also the title for the chat is currently generated wrong - the same model is used by default as for generation, so the title begins with "<thinking>". Be that as it may, I think these will be the first "locally usable" reasoning models for me. URL for the checkpoints: https://ollama.com/library/deepseek-r1
It's remarkable how effectively China is salting the earth for OpenAI, Meta, Anthropic, Google, and X.ai with a small fraction of those companies compute capacity. Sanctions tend to backfire in unpredictable ways sometimes. Reasoning models aside, you can get a free GPT 4o - grade chatbot at chat.deepseek.com and it actually runs faster. Their API prices are much lower as well. And they disclose the living Confucius out of their methods in their technical reports. Kudos!
Tried this yesterday under Ollama. Asked it to explain and implement bitonic sort. It churned for a little bit, and started putting together an implementation towards the end, but then seemingly ran out of generation window because the CoT was so long winded. TL;DR: promising, but needs more work. I'm sure they'll improve it greatly in the months to come, the potential is pretty clearly there.
They're salting the earth / destroying the moat for OpenAI and Anthropic, and to a lesser extent, for Google. Basically, now pretty much anyone can do "sufficient" AI right in their garage. I use Mistral Large, and most of the time its output is easily on par with GPT4, with total privacy. This destruction of the moat prevents the future where e.g. OpenAI becomes your interface to the internet, recommendations, shopping, social media, news, etc. The moment you start doing something that's clearly valuable, 10 other companies pop up and destroy your potential future margin. They are trying the "internet search" part of that, because that needs a search index, and that's not something that's easy to do. We're lucky that Google blew its AI lead so badly - it _could_ realistically become all of the above, if it wasn't so badly mismanaged.
I don’t doubt it at all. I’ve been to a couple of countries the US propagandizes heavily against (China and Russia) and what you hear about them in the US is almost entirely horseshit. My expectation is NK and Iran will be similar.
Yes, I’m aware. But that can’t be what most of the population in Iran thinks, since some of the Persians I know in the US support a full scale war and regime change in Iran, while acknowledging that hundreds of thousands of their country people would die as a result and chance of success is near zero. More disturbingly they also seem to suggest that people outside Tehran and other large cities should have no say in how Iran is run, and when I say there’s no way they’ll allow that, there’s a blank stare. Also BBC is not a reliable source for information about either country. It’s quite literally government affiliated media.
Ollama also exposes an RPC interface on Linux at least. That can be used with open-webui. Maybe llama.cpp has this, idk, but I use Ollama mostly through its RPC interface. It’s excellent for that.
There are pay caps that do not correspond to reality, and you often don’t need such capable people for 40+ hours a week. When I consulted I explicitly told my clients that they do not need me full time, and I will have two clients add spend half of my time each week on each. I billed each client half my full time amount. Whether their accounting department knew all that is none of my business.