lambda · HackerLangs

قبل 12 ساعةdiscuss

Yeah, Kraken SDR removed some functionality due to these concerns, if I remember correctly.

Odd, because export controls don't generally apply to published material (like open source software), but maybe they were worried that because they were also selling the hardware they could have issues due to the combo being export controlled.

Ah, found discussion of what exactly it was they pulled, it was the passive radar code: https://www.reddit.com/r/RTLSDR/comments/yu9rei/krakenrf_pul...

And indeed, they confirmed that they believe the open source software should be OK, but they had concerns because they also sell the compatible hardware: https://nitter.net/rtlsdrblog/status/1591657740229046274

lambda

قبل 12 ساعةdiscuss

But there are already benchtop or handheld signal analyzer for that purpose.

This seems more like a tool for checking across entire large assemblies like an entire building, car, aircraft, etc, for unknown sources. If you have an individual discrete device that you're already testing, just using traditional instrumentation seems reasonable, but on a large, complex assembly, I can see it being useful. Also useful for things like detecting if a particular antenna is working without actually going up there to measure near it; if you have a MIMO setup with multiple antennas, this might make it easier to check if all of them are working correctly when mounted in inconvenient areas.

lambda

قبل 12 ساعةdiscuss

I think that for a single device, this probably wouldn't help much over just having a more traditional signal analyzer, either benchtop or handheld. If you know what you're testing, just using a signal analyzer around it will give you a good first pass picture of emissions, and probably be much more informative and precise than this.

This seems more useful for finding unknown or hidden RF sources, for instance looking thorugh an entire building to find unknown RF sources, or maybe a whole complex assembly like a car or aircraft.

lambda

قبل 4 أيامdiscuss

This is really weird. Someone creating 4 new accounts just to call this project fraudulent because it can't statically analyze every property you'd like? Does this person have a personal grudge against the author, or something?

lambda

قبل 9 أيامdiscuss

Tried it out. I'm compring against Qwen 3.5 122B-A10B, so a much larger model. It gets some correct, but Qwen 3.5 122B-A10B has done much better. Gemma 4 12B even hallucinated some species in trying to identify a plant, and the other guesses it made weren't all that close, while Qwen 3.5 122B-A10B got it right on the first try.

12B did get one right that 31B got wrong. I'd have to do a much more thorough eval to really compare, just a few anecdotal observations and it's kind of hard to really distinguish, but from the samples I've seen, Qwen 3.5 122B-A10B is doing much better at this task.

The 12B architecture definitely is interesting, and it may punch above its weight due to this (though again, would really need to do proper evals to compare). But of the models I've tried, Qwen3.5 122B-A10B really seems like the best for this kind of task.

lambda

قبل 11 يومًاdiscuss

I haven't run a proper eval, but I've been getting better luck with Qwen models than Gemma on plant and animal identification using vision.

I do like Gemma for translation, however.

lambda

قبل 21 يومًاdiscuss

How could the harness fix this? It's the jinja template used by the inference engine to render the API requests into the raw text that gets tokenized and completed by the model. Unless you're using something like the raw completions API instead of the `/v1/chat/completions` API, and effectively applying the template yourself. In which case, you could also just modify the jinja template on your server.

Anyhow, I've heard mixed results on any method of supplying reasoning traces beyond the current turn to models not trained on them. For some models, I've heard that it works fine this way, for others I've heard it degrades performance. But I don't know of anyone who has any kind of reliable benchmark for how well this works.

lambda

قبل 24 يومًاdiscuss

Much more complex than that. Even if it does give you a speedup at certain tasks, is it worth the cost and risks? You go faster, but now you have more code that you don't understand and so won't be as good at maintaining. There's the engergy use, the water use, the scrapers destroying the internet, the massive piles of slop, the hallucinations and bullshit, etc.

lambda

قبل 24 يومًاdiscuss

It means that even if it works for certain tasks, I think that the problems caused by use of LLMs outweigh their benefits. I think it's a bad idea to generate large piles of code that you don't understand, but due to competitive pressures, it's too tempting for people to pass up, leading to a world in which software is getting worse by the day, while pumping CO2 into the atmosphere and boiling scarce water supplies to do so, DDOSing websites to scrape the data, and polluting the internet with mountains of slop.

This isn't about using rice cookers or not, that's a personal choice for how you cook your food, and choosing to do so or not really only affects the person cooking and cleaning. A rice cooker probably uses a similar amount of energy as cooking it by hand, possibly even less.

But when people using LLMs are causing active harm, and are making it more difficult to collaborate on a team, it's a lot harder to accept that it's just a personal preference.

If you wanted to use the rice cooker analogy, imagine if rice cookers let you cook rice in just one minute. Faster, don't have to wait for the rice to be done, great! But in order to do so, you have to cook 50 pounts of rice, but throw out the majority of it, and use a thousand kilowatt hours of energy to do so. You'd better believe I'm going to be skeptical of everyone deciding that they suddenly have to use these 1-minute rice cookers that burn so much energy and generate so much waste.

lambda

قبل 24 يومًاdiscuss

Huh? There is a Claude 4 Opus. It was released about a year ago. It is retired by now, in fact, just retired yesterday: https://platform.claude.com/docs/en/about-claude/model-depre...

But it is still available on Google Vertex according to OpenRouter (though it's possible that info is just out of date, it's currently quoting 3tps which is unusably slow): https://openrouter.ai/anthropic/claude-opus-4

lambda

قبل 25 يومًاdiscuss

Not a harness issue. The harness (pi in my case) passes back the cot for all previous turns.

The jinja template is what renders the openai-format request sent by the harness, into the actual string of text that will be tokenized and fed to the model. For models without preserve thinking support, the jinja template drops the reasoning from all but the current turn.

Here is the default jinja for Gemma 4: https://huggingface.co/google/gemma-4-31B-it/blob/main/chat_...

    {#- Render reasoning/reasoning_content as thinking channel -#}
    {%- set thinking_text = message.get('reasoning') or message.get('reasoning_content') -%}
    {%- if thinking_text and loop.index0 > ns_turn.last_user_idx and message.get('tool_calls') -%}
        {{- '<|channel>thought\n' + thinking_text + '\n<channel|>' -}}
    {%- endif -%}

You see that it only preserves the thinking for indexes that are later than the last user message; thinking is only preserved for a single turn (which can include a lot of interleaved thinking and tool calls), once it goes back to the user and the user replies, it will replay the tool calls but not the thinking between them.

Here's Qwen 3.6 by comparison: https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/chat_t...

        {%- if (preserve_thinking is defined and preserve_thinking is true) or (loop.index0 > ns.last_query_index) %}
            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
        {%- else %}
            {{- '<|im_start|>' + message.role + '\n' + content }}
        {%- endif %}

It additionally has a preserve_thinking flag that you can set. If that's set, it will include all turns thinking in the text passed to the model. But you do have to set that, it's not the default.

It's possible to modify the jinja file that you're using with a model. Some people do that with models that haven't been specifically trained for it, and report good results; but some report that because it wasn't trained for that, they get worse results if they include thinking from previous turns.

So for models like Gemma, you would have to modify the default jinja to enable this. For Qwen, you can just set the preserve_thinking flag to get this behavior; and apparently they have trained in this mode so you get better results than models that have not trained this way.

lambda

قبل 25 يومًاdiscuss

Right. Local models haven't quite hit that level yet. The biggest open models, which you need tens of thousands of dollars of hardware to run at reasonable speed, have pretty much hit that level of capability, but most models you can reasonably run at home aren't quite there yet. But given the gap, if local models keep improving, you'd expect to maybe see that level by this November.

lambda

قبل 25 يومًاdiscuss

Which Opus?

Anthropic has been releasing models named Opus since 2024 with Claude 3 Opus.

Opus has gotten vastly more capable since then.

Local model far surpass Opus 3. They even surpass Opus 4 on most benchmarks.

Sure, if you compare to the latest Opus 4.8 or even 4.6, they're not there yet. But there's a huge difference in performance between 4 and 4.8.

lambda

قبل 25 يومًاdiscuss

So, one of the ways that this problem manifests is that most local models aren't trained on preserving the full reasoning between turns. Every turn, they skip passing the reasoning trace from previous turns to the the LLM. So if on one turn you have a long interleaved chain of reasoning and tool calls, then it responds to you, and then you give a new prompt to fix something, it has to re-process all of those tools calls now with the reasoning stripped out.

Qwen 3.6 has finally been trained both with and without preserving thinking, so you can optionally enable preserving thinking. This will use up a bit more context, but it will avoid having to do this re-processing of long agentic turns, and also the preserved thinking can avoid having to re-do some of the same reasoning over again in later turns.

Besides that, modern LLMs don't only use full attention (apparently, attention is not all you need). Full attention is very expensive to compute and store (0(n^2)). But additionally, full attention is actually bad at certain kinds of reasoning; keeping track of some value that gets replaced over the course of time, for example. So most models these days use various forms of local attention which is fixed length and gets updated as you go; sliding window attention, Mamba-2 state space models, etc.

But one advantage of attention is that you can go back and reprocess by truncating the KV cache and starting over. You can't do that with other forms of local attention; you've lost the state earlier in the sequence.

So to allow you to go back without fully recomputing the cache all over again, your engine will save snapshots of the local attention state at various times, so if you need to go back to recompute the cache, you can start from the last snapshot. However, these snapshots can get large, you can't keep too many of these, so sometimes you need to go back quite far to get to one, or they're all past the point you need to go back to and you need to start over again from the beginning.

There have been particular bugs in llama.cpp that have caused this to be triggered more often than it should; for instance, it wouldn't take snapshots before turns that included images at one point, so if you had an image heavy agentic workflow, that issue plus the lack of preserving thinking would mean you would frequently have to go back and start over from scratch.

Some of these issue have been fixed, some are addressed by preserving thinking. There are still some issues sometimes; for instance, one that's hard to fix is that the tokens generated autoregressively don't always parse the same when doing prefill. For instance, you could generate something as two tokens "pre" and "fill", but it turns out that "prefill" is also a single token so the tokenizer will use that, so when you send that back again on the next turn, it will see a divergence and have to recompute from that point. It might be possible to ignore that and use the not fully greedy tokenization that's in the cache, but I've definitely seen llama.cpp have to do some cache recomputation due to that.

lambda

قبل 25 يومًاdiscuss

Yeah, definitely.

lambda

قبل 25 يومًاdiscuss

Yeah. I was pointing out that local 3b active models outperform frontier models from a year ago.

Will this trend continue? Who knows. Both the frontier and local model will probably continue to get better. Which one will hit the top of the S-curve first? Hard to say, really. But what you can do right now locally is better than what you could do a year ago on the frontier, and lots of people were already using it pretty heavily a year ago.

Hoever, November is when most folks agree that the frontier models got good enough for much of their work. Local models aren't quite there yet (where by "local" I mean "can run at reasonable speed and quant on a system less that $10,000 with today's RAM and GPU prices"). The biggest open weights models are getting there, but those require something like an 8x H100 server to reasonably run.

It's likely that there will always be a gap between frontier and local if you're comparing models at the same time, you can just do a lot more with terabytes of HBM than gigabytes of DDR. But will local models get good enough to be usable for useful work? For many folks, they already are.

lambda

قبل 25 يومًاdiscuss

OK, it looks like he did a browser OS test with both Claude 4 Opus and Qwen 3.6 35B-A3B.

Claude 4 Opus: https://youtu.be/J7omabtqnBM?t=193

Qwen 3.6 35B A3B: https://youtu.be/gVU-DQeqkI0?t=215

Qwen 3.6 produced far more working functionality than Claude 4 Opus did.

Obviously, just one test of a single one-shot prompt of a silly toy OS, but yeah, this particular test shows Qwen 3.6 running locally dramatically outperforming Claude 4 Opus, which was a frontier model a year ago.

lambda

قبل 25 يومًاdiscuss

Which Opus? They certainly outperform Claude 3 Opus.

Anyhow, feel free to try them out head to head on OpenRouter. I'd love to see someone write up their results, of a modern local sized open source model vs. frontier models from ~a year ago, on something other than the standard benchmarks.

lambda

قبل 25 يومًاdiscuss

That should do pretty well. Memory bandwidth is the biggest bottleneck for token generation, at 644 GB/s you should be able to do pretty well on a 9070, while prompt proessing is more compute bound and Nvidia tends to have the edge there.

16 GiB won't fit you much, so you'd probably want at least 2x, and preferably 3x of those, and then you need a motherboard, power, etc. that can handle that.

lambda

قبل 25 يومًاdiscuss

I use Vulkan mostly instead of ROCm. Vulkan is actually a bit faster, paradoxically. I do switch out and try them both out, and it's not a huge difference, but I've been mostly saying on Vulkan.

The re-processing context every turn problem is definitely something I've hit. Some of the causes have been solved upstream in llama.cpp; make sure you're up to date.

But another cause of the issue that has a big effect is that older Qwen models didn't support preserving thinking. This means that each time you have a long sequence of tool calls with interleaved thinkging, as soon as you had your next turn in the chat, it would have to re-process all of that as it would drop all of the reasoning.

Qwen 3.6, however, now supports preserving thinking. This can use a bit more context, becasue you're not dropping the thinking every turn, but it re-uses the cache better, not causing you to have to reprocess a whole turn at a time each time.

In my models.ini, I have this for the Qwen3.6 models:

  chat-template-kwargs = {"preserve_thinking": true}

There are still occasional issues I hit where it will have to re-process, but getting up to date and enabling preserve_thinking has helped a ton.