HackerTrans
TopNewTrendsCommentsPastAskShowJobs

josefcub

16 karmajoined vor 4 Jahren

comments

josefcub
·vorgestern·discuss
Well, I ran a couple of experiments a couple years ago against a 10.7b SOLAR-based language model and MUDs. What I found is that dumping one into a MUD that had been built specifically for humans resulted in a lot of confusion that usually ended up with the model looping around in a circle looking for something or someone to interact with.

When I repeated the experiment with a MUD that I'd built by hand (A small American town) for the LLM's own limitations (Descriptions referenced things that I made sure existed, more common verbs existed for it to use on things, there was a map facility, and at least me to interact with on a second connection), I found the agent much more likely to take its time exploring, making up its own goals, and spending time traveling in the space just communicating with me in a roleplaying context.

It was an interesting time; I wasn't sure what I was expecting it to do after the first experiment, but it seemed to really jump into the second one and kept playing until I terminated the experiment.

If I were going to do it a third time, I'd probably create objects and give a modern agent fetch quests and other goals, and see how well it independently can handle that.
josefcub
·vor 18 Tagen·discuss
I've never used such a thing. If the intent is to get the LLM to not perform reasoning before it answers, most of the APIs have a toggle that lets you minimize or stop thinking.

I'd really like to understand why you'd use "Answer now" though; what's your use-case?
josefcub
·vor 2 Monaten·discuss
Hermes Agent is really decent.

Don't rethink your entire company, offer your customers choice instead. Some may prefer OpenClaw and others might be better served with Hermes Agent or even a third type. If you can find the best use-cases for each, you can help customers decide what they need when they buy your service and get all of the customers instead of a subset.
josefcub
·vor 2 Monaten·discuss
I'd started using DOS-based machines in the 80s, and ended up with a copy of WordStar 3.3. I used it so much as a kid that I still have the WordStar keybindings stuck in my muscle memory.

There's one CLI text editor left with WordStar keybindings used natively: Joe's Own Editor. If you're like me and enjoy WordStar's key combinations and the editing paradigm, JOE is a good text editor to use on the command line.
josefcub
·vor 2 Monaten·discuss
Brave's "Leo" AI is configurable enough to specify local endpoints for processing, instead of going wherever they want it to go. I've set it up to use my own systems, and it works just fine like that.

If you have a beefy enough device, then yes this can be done on-device.
josefcub
·vor 3 Monaten·discuss
I've got 256GB of RAM on a Mac Studio M3 Ultra. Other posters are right: The M3 Ultra's prefill is super slow with really large models, 3-5 minutes while it digests the new additions to its context before it continues. On my heavy RAM model, I _can_ run 400b-500b models at Q2, and up to about 750b models at Q1, but the wait isn't the worst part.

Lower quants like that affect its output, making it less capable overall and letting it easily forget things.

Here's what I'd do with 96GB of RAM: Run Qwen 3.6 35b-a3b at Q8 for coding/agentic tasks. You'll get around 70tokens generated per second, the prefill is lightning fast in comparison, and you'll get a lot of work done. Qwen 3.6 27b is out now too, and I'm getting 17tok/sec token generation with a slower prefill.

The upshot is that you'll still have 20-40GB of RAM left for your workstation and development loads. Running Qwen 3.6 35b or 27b at Q8 quantization, the model at 128k context uses about 40GB of RAM; my OS and application load uses 20-30GB most of the time, for a total of 60-70. That's plenty of room in memory for you to work _and_ run inference.

You _may_ end up getting Deepseek 4 Flash running, but it'll be a lower quantization like Q2 or Q3, making it kind of dumb in comparison. And you may not have enough memory left over for any appreciable amount of context. Working with today's reasoning models needs context for it to generate and give out good answers. Doubly so for agentic/coding tasks.
josefcub
·vor 3 Monaten·discuss
My google-fu is failing me at the moment to cite sources, but here's an example ~/.config/crush/crush.json file (based on my own) showing the options to remove telemetry and provider auto updates, and the connection info to connect to a localhost model on an OpenAI-compatible endpoint:

{ "$schema": "https://charm.land/crush.json", "options": { "disable_provider_auto_update": true, "disable_metrics": true }, "providers": { "ollama": { "name": "Local Models", "base_url": "http://localhost:11434/v1", "api_key": "nunya", "type": "openai-compat", "models": [ { "name": "Qwen 3.5 Local", "id": "qwen-3.5-35b-planning", "cost_per_1m_in": 0.01, "cost_per_1m_out": 0.01, "context_window": 131072, "think": true, "default_max_tokens": 5120, "supports_attachments": true } ] } } }

...or not, thanks to formatting. I can't even search for help formatting this text box, because of HN's nature haha
josefcub
·vor 3 Monaten·discuss
Try charmbracelet's crush, found here:

https://github.com/charmbracelet/crush

Crush is pretty new, but getting better all the time. It's written in Go, so no node hijinks to get it working. It works fine with my ollama or llama-server localhost endpoints, and I've used it to make up a couple of internal projects without any issues.

It does have internal telemetry and such (including updating its list of external models it can use) that can be turned off in the crush.json configuration file.

If you're on a Mac, you can install via homebrew or use the more traditional route via Github.
josefcub
·vor 5 Monaten·discuss
I am! I moved from a shoebox Linux workstation with 32MB of RAM and a 12GB RTX 3060 to a 256GB M3 Ultra, mainly for unified memory.

I've only had it a couple of months, but so far it's proving its worth in the quality of LLM output, even quantized.

I generally run Qwen3-vl at 235b, at a Q4_K_M quantization level so that it fits, and it leaves me plenty of RAM for workstation tasks while delivering tokens at around 30tok/s

The smaller Qwen3 models (like qwen3-coder) I use in tandem, of course they run much faster and I tend to run them at higher quants up to Q8 for quality purposes.

The gigantic RAM's biggest boon, I've found, is letting me run the models with full context allocated, which lets me hand them larger and more complicated things than I could before. This alone makes the money I spent worth it, IMO.

I did manage to get glm-4.7 (a 358b model) running at a Q3 quantization level; it's delivery is adequate quality-wise, although it delivers at 15tok/s, though I did have to cut down to only 128k context to leave me enough room for the desktop.

If you get something this big, it's a powerhouse, but not nearly as much of a powerhouse as a dedicated nVidia GPU rig. The point is to be able to run them _adequately_, not at production speeds, to get your work done. I found price/performance/energy usage to be compelling at this level and I am very satisfied.
josefcub
·vor 9 Monaten·discuss
I think this is fun! I ended up with 92/102, because a few were non-obvious. If I had a critique it's that the AI entries were repeated on multiple occasions, making the decision between them trivial.

If you can find some public domain literature from the 20th century, it would be a much harder game.